Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Conda package isn't build with GDS support! #378

Closed
madsbk opened this issue May 14, 2024 · 14 comments · Fixed by #379 or #381
Closed

Conda package isn't build with GDS support! #378

madsbk opened this issue May 14, 2024 · 14 comments · Fixed by #379 or #381
Labels
bug Something isn't working

Comments

@madsbk
Copy link
Member

madsbk commented May 14, 2024 •
edited
Loading

Reproducer

mamba create -n kvikio-env -c rapidsai-nightly -c conda-forge python=3.11 cuda-version=12.2 kvikio
conda activate kvikio-env
pip install dask

# Run the benchmark with compatibility mode disabled, which forces KvikIO to use cuFile
KVIKIO_COMPAT_MODE=off python python/benchmarks/single-node-io.py

Roundtrip benchmark
----------------------------------
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
   WARNING - cuFile compat mode   
         GDS not enabled          
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
GPU               | Unknown (install pynvml)
GPU Memory Total  | Unknown (install pynvml)
BAR1 Memory Total | Unknown (install pynvml)
GDS driver        | N/A (Compatibility Mode)
GDS config.json   | /etc/cufile.json
----------------------------------
nbytes            | 10485760 bytes (10.00 MiB)
4K aligned        | True
pre-reg-buf       | True
directory         | /tmp/tmp71d24zvz
nthreads          | 1
nruns             | 1
==================================
Traceback (most recent call last):
  File "/home/mkristensen/repos/kvikio/python/benchmarks/single-node-io.py", line 429, in <module>
    main(args)
  File "/home/mkristensen/repos/kvikio/python/benchmarks/single-node-io.py", line 326, in main
    read, write = API[api](args)
  File "/home/mkristensen/repos/kvikio/python/benchmarks/single-node-io.py", line 47, in run_cufile
    res = f.write(data)
  File "/home/mkristensen/apps/miniforge3/envs/kvikio-env2404/lib/python3.10/site-packages/kvikio/cufile.py", line 261, in write
    return self.pwrite(buf, size, file_offset, task_size).get()
  File "/home/mkristensen/apps/miniforge3/envs/kvikio-env2404/lib/python3.10/site-packages/kvikio/cufile.py", line 33, in get
    return self._handle.get()
  File "libkvikio.pyx", line 25, in kvikio._lib.libkvikio.IOFuture.get
RuntimeError: KvikIO not compiled with cuFile.h

The latest version that works is 24.02, the issue must have started somewhere between version 24.02 and 24.04.

cc. @aslobodaNV
@madsbk madsbk changed the title Conda package of kvikio isn't build GDS support! Conda package of kvikio isn't build with GDS support! May 14, 2024
@madsbk madsbk changed the title Conda package of kvikio isn't build with GDS support! Conda package isn't build with GDS support! May 14, 2024
@madsbk madsbk added the bug Something isn't working label May 14, 2024
@jakirkham
Copy link
Member

Looking at a recent CI build, am seeing the following:

  -- Found cuFile: /opt/conda/conda-bld/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/libcufile.so
  -- Found CUDAToolkit: /opt/conda/conda-bld/_build_env/targets/x86_64-linux/include (found version "12.2.140")
  -- Found cuFile Batch API: TRUE
  -- Found cuFile Stream API: TRUE

So it does seem to detect it. This is the C++ build

Currently the Python build doesn't share enough info about its configuration to tell

@KyleFromNVIDIA
Copy link
Contributor

KvikIO's driver.hpp has two different inline class definitions based on the value of KVIKIO_CUFILE_FOUND:

kvikio/cpp/include/kvikio/driver.hpp

Line 43 in 6ed7bcc

#ifdef KVIKIO_CUFILE_FOUND

kvikio/cpp/include/kvikio/driver.hpp

Lines 190 to 272 in 6ed7bcc

#else
struct DriverInitializer {
// Implement a non-default constructor to avoid `unused variable` warnings downstream
DriverInitializer() {}
};
struct DriverProperties {
// Implement a non-default constructor to avoid `unused variable` warnings downstream
DriverProperties() {}
static bool is_gds_available() { return false; }
[[nodiscard]] static unsigned int get_nvfs_major_version()
{
throw CUfileException("KvikIO not compiled with cuFile.h");
}
[[nodiscard]] static unsigned int get_nvfs_minor_version()
{
throw CUfileException("KvikIO not compiled with cuFile.h");
}
[[nodiscard]] static bool get_nvfs_allow_compat_mode()
{
throw CUfileException("KvikIO not compiled with cuFile.h");
}
[[nodiscard]] static bool get_nvfs_poll_mode()
{
throw CUfileException("KvikIO not compiled with cuFile.h");
}
[[nodiscard]] static std::size_t get_nvfs_poll_thresh_size()
{
throw CUfileException("KvikIO not compiled with cuFile.h");
}
static void set_nvfs_poll_mode(bool enable)
{
throw CUfileException("KvikIO not compiled with cuFile.h");
}
static void set_nvfs_poll_thresh_size(std::size_t size_in_kb)
{
throw CUfileException("KvikIO not compiled with cuFile.h");
}
[[nodiscard]] static std::vector<CUfileDriverControlFlags> get_nvfs_statusflags()
{
throw CUfileException("KvikIO not compiled with cuFile.h");
}
[[nodiscard]] static std::size_t get_max_device_cache_size()
{
throw CUfileException("KvikIO not compiled with cuFile.h");
}
static void set_max_device_cache_size(std::size_t size_in_kb)
{
throw CUfileException("KvikIO not compiled with cuFile.h");
}
[[nodiscard]] static std::size_t get_per_buffer_cache_size()
{
throw CUfileException("KvikIO not compiled with cuFile.h");
}
[[nodiscard]] static std::size_t get_max_pinned_memory_size()
{
throw CUfileException("KvikIO not compiled with cuFile.h");
}
static void set_max_pinned_memory_size(std::size_t size_in_kb)
{
throw CUfileException("KvikIO not compiled with cuFile.h");
}
[[nodiscard]] std::size_t get_max_batch_io_size()
{
throw CUfileException("KvikIO not compiled with cuFile.h");
}
};
#endif

The case where it's not defined throws the error we're seeing:

kvikio/cpp/include/kvikio/driver.hpp

Line 204 in 6ed7bcc

throw CUfileException("KvikIO not compiled with cuFile.h");

CMake defines KVIKIO_CUFILE_FOUND appropriately:

kvikio/cpp/CMakeLists.txt

Line 161 in 6ed7bcc

target_compile_definitions(kvikio::kvikio INTERFACE KVIKIO_CUFILE_FOUND)

However, the Cython code bypasses anything built by CMake and includes driver.hpp directly:

kvikio/python/kvikio/_lib/kvikio_cxx_api.pxd

Line 36 in 6ed7bcc

cdef extern from "<kvikio/driver.hpp>" namespace "kvikio" nogil:

So it seems to me that we need to tell Cython to define KVIKIO_CUFILE_FOUND, but only if it's defined in the C++ code.

@KyleFromNVIDIA
Copy link
Contributor

I can see now that KvikIO's exported CMake code is already setting this if cuFile is installed:

kvikio/cpp/CMakeLists.txt

Lines 132 to 169 in 6ed7bcc

get_property(already_set_kvikio DIRECTORY PROPERTY kvikio_already_set_defines SET)
if(NOT already_set_kvikio)
set_property(DIRECTORY PROPERTY kvikio_already_set_defines "ON")
# Find cuFile and determine which features are supported
find_package(cuFile QUIET)
if(NOT cuFile_FOUND)
message(WARNING "KvikIO: cuFile not found")
else()
file(READ "${cuFile_INCLUDE_DIRS}/cufile.h" CUFILE_H_STR)
string(FIND "${CUFILE_H_STR}" "cuFileBatchIOSetUp" cuFileBatchIOSetUp_location)
if(cuFileBatchIOSetUp_location EQUAL "-1")
set(cuFile_BATCH_API_FOUND FALSE)
else()
set(cuFile_BATCH_API_FOUND TRUE)
endif()
message(STATUS "KvikIO: Found cuFile Batch API: ${cuFile_BATCH_API_FOUND}")
string(FIND "${CUFILE_H_STR}" "cuFileReadAsync" cuFileReadAsync_location)
if(cuFileReadAsync_location EQUAL "-1")
set(cuFile_STREAM_API_FOUND FALSE)
else()
set(cuFile_STREAM_API_FOUND TRUE)
endif()
message(STATUS "KvikIO: Found cuFile Stream API: ${cuFile_STREAM_API_FOUND}")
endif()
# Enable supported cuFile features in KvikIO
if(cuFile_FOUND)
target_link_libraries(kvikio::kvikio INTERFACE cufile::cuFile_interface)
target_compile_definitions(kvikio::kvikio INTERFACE KVIKIO_CUFILE_FOUND)
if(cuFile_BATCH_API_FOUND)
target_compile_definitions(kvikio::kvikio INTERFACE KVIKIO_CUFILE_BATCH_API_FOUND)
endif()
if(cuFile_STREAM_API_FOUND)
target_compile_definitions(kvikio::kvikio INTERFACE KVIKIO_CUFILE_STREAM_API_FOUND)
endif()
endif()
endif()

So we need to make sure cuFile is installed when the Python Conda package is built.

@KyleFromNVIDIA KyleFromNVIDIA mentioned this issue May 14, 2024
Merged
@vyasr
Copy link
Contributor

vyasr commented May 15, 2024

The same error is also present in the cufile shim code, so it could be coming from there and not driver.hpp. Note that it is also possible for the version of is_gds_enabled in the cufile-enable branch of driver.hpp to return false based on values produced by the cufile shim. Given that adding the libcufile package to the build env hasn't worked in #379 yet, those might be some additional things to check.

@madsbk
Copy link
Member Author

madsbk commented May 15, 2024

Notice, is_gds_enabled() == false should not trigger the RuntimeError: KvikIO not compiled with cuFile.h error. It just means that cuFile will not use GDS acceleration.

@rapids-bot rapids-bot bot closed this as completed in #379 May 15, 2024
rapids-bot bot pushed a commit that referenced this issue May 15, 2024
@KyleFromNVIDIA
Move cuFile linking to kvikio target (#379)
4355c29 
The exported `kvikio` target has an `INTERFACE` dependency on cuFile. Add this dependency in the local buildsystem too for consistency, and remove the usage from the examples. This will allow all targets that link against `kvikio` to automatically get cuFile, including the Python modules.

Fixes: #378

Authors:
  - Kyle Edwards (https://github.com/KyleFromNVIDIA)

Approvers:
  - Bradley Dice (https://github.com/bdice)
  - Robert Maynard (https://github.com/robertmaynard)

URL: #379
@jakirkham
Copy link
Member

Am going to reopen so we can track this fix as well

xref: #381

@jakirkham jakirkham reopened this May 15, 2024
@jakirkham jakirkham mentioned this issue May 15, 2024
Merged
@rapids-bot rapids-bot bot closed this as completed in #381 May 16, 2024
rapids-bot bot pushed a commit that referenced this issue May 16, 2024
@KyleFromNVIDIA
Fix conda recipe for kvikio (#381)
261445b 
Use `build.sh` in `meta.yaml`, and pass `-DFIND_KVIKIO_CPP=ON` to CMake.

xref: #379 (comment)

Fixes #378

Authors:
  - Kyle Edwards (https://github.com/KyleFromNVIDIA)

Approvers:
  - Bradley Dice (https://github.com/bdice)
  - Vyas Ramasubramani (https://github.com/vyasr)
  - Jake Awe (https://github.com/AyodeAwe)

URL: #381
@fstrug
Copy link

fstrug commented Oct 22, 2024

Hi, I think I might be running into a similar related error. I am installing kvikio with Conda and I can see the libcufile is installed in my environment. When I try to run the benchmark:

> KVIKIO_COMPAT_MODE=off python python/kvikio/kvikio/benchmarks/single_node_io.py 

Roundtrip benchmark
----------------------------------
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
   WARNING - cuFile compat mode   
         GDS not enabled          
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
GPU               | Unknown (install nvidia-ml-py)
GPU Memory Total  | Unknown (install nvidia-ml-py)
BAR1 Memory Total | Unknown (install nvidia-ml-py)
GDS driver        | N/A (Compatibility Mode)
GDS config.json   | /etc/cufile.json
----------------------------------
nbytes            | 10485760 bytes (10.00 MiB)
4K aligned        | True
pre-reg-buf       | False
directory         | /tmp/tmp4__41g_o
nthreads          | 1
nruns             | 1
==================================
Traceback (most recent call last):
  File "/home/fstrug/kvikio/python/kvikio/kvikio/benchmarks/single_node_io.py", line 375, in <module>
    main(args)
  File "/home/fstrug/kvikio/python/kvikio/kvikio/benchmarks/single_node_io.py", line 283, in main
    read, write = API[api](args)
                  ^^^^^^^^^^^^^^
  File "/home/fstrug/kvikio/python/kvikio/kvikio/benchmarks/single_node_io.py", line 45, in run_cufile
    f = kvikio.CuFile(file_path, flags="w")
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fstrug/.conda/envs/img_cuda12.2-kvikio/lib/python3.12/site-packages/kvikio/cufile.py", line 88, in __init__
    self._handle = file_handle.CuFile(file, flags)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "file_handle.pyx", line 97, in kvikio._lib.file_handle.CuFile.__init__
RuntimeError: cuFile error at: /home/fstrug/.conda/envs/img_cuda12.2-kvikio/include/kvikio/file_handle.hpp:196: internal error

When trying to cmake kvikio/cpp/examples/downstream/, it is unable to find libcufile even though it is installed within the Conda environment.

(img_cuda12.2-kvikio) [fstrug@jupyter-fstrug build]$ cmake ..
-- The CXX compiler identification is GNU 11.4.1
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Downloading CPM.cmake to /home/fstrug/kvikio/cpp/examples/downstream/build/cmake/CPM_0.35.5.cmake
-- CPM: adding package [email protected] (branch-24.12)
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- CPM: KvikIO: using local package [email protected]
-- Unable to find cublas_v2.h in either "/opt/conda/targets/x86_64-linux/include" or "/opt/conda/math_libs/include"
-- Found CUDAToolkit: /opt/conda/targets/x86_64-linux/include (found version "12.2.140") 
-- Could NOT find cuFile (missing: cuFile_INCLUDE_DIR) 
CMake Warning at build/_deps/kvikio-src/cpp/CMakeLists.txt:71 (message):
  Cannot find cuFile - KvikIO will still work but won't use GPUDirect Storage
  (GDS)


-- CPM: KvikIO: using local package [email protected]
-- CPM: KvikIO: using local package [email protected]
-- Configuring done (4.6s)
-- Generating done (0.0s)
-- Build files have been written to: /home/fstrug/kvikio/cpp/examples/downstream/build

@jakirkham
Copy link
Member

Can you please share conda info and conda list?

@fstrug
Copy link

fstrug commented Oct 22, 2024

Here are the outputs.

conda_list.txt
conda_info.txt

@jakirkham
Copy link
Member

Thanks fstrung! 🙏

From a quick pass, those look reasonable

In the build case expect we need to pass more arguments to cmake like

cmake .. \
    -DCMAKE_PREFIX_PATH="${CONDA_PREFIX}" \
    -DCMAKE_INSTALL_PREFIX:PATH="${CONDA_PREFIX}" \
    ...
;

Though we do have build.sh, which may help with that. So one could do the following to build and install the C++ and Python packages into your Conda env (the install location can also be customized)

./build.sh libkvikio kvikio

To your bigger question, think we should track this in a new issue. It certainly could be a packaging issue or it could also be some other library issue ( for example: #497 ). Think we need to dive deeper. Would you be alright opening a new issue with the info you have provided here?

As a side note, it is possible to dump all the text info into it with using the pattern below, which may help as more folks look into it

<details>

```bash
# Lots of output here
\```

</details>

@madsbk
Copy link
Member Author

madsbk commented Oct 23, 2024 •
edited
Loading

The following fresh conda env with kvikio v12.10 works for me:

name: rapids-24.10
channels:
  - rapidsai
  - rapidsai-nightly
  - conda-forge
  - nvidia
  - pytorch
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=2_gnu
  - aiohappyeyeballs=2.4.3=pyhd8ed1ab_0
  - aiohttp=3.10.10=py312h178313f_0
  - aiosignal=1.3.1=pyhd8ed1ab_0
  - anyio=4.6.2.post1=pyhd8ed1ab_0
  - aom=3.9.1=hac33072_0
  - argon2-cffi=23.1.0=pyhd8ed1ab_0
  - argon2-cffi-bindings=21.2.0=py312h66e93f0_5
  - arrow=1.3.0=pyhd8ed1ab_0
  - asciitree=0.3.3=py_2
  - attrs=24.2.0=pyh71513ae_0
  - aws-c-auth=0.7.31=hdfc988b_4
  - aws-c-cal=0.7.4=h3f62d97_3
  - aws-c-common=0.9.30=hb9d3cd8_0
  - aws-c-compression=0.2.19=hc418e29_3
  - aws-c-event-stream=0.4.3=h3c86c75_6
  - aws-c-http=0.8.10=h2ee613e_4
  - aws-c-io=0.14.20=h9a4becb_1
  - aws-c-mqtt=0.10.7=ha9d9be6_4
  - aws-c-s3=0.6.7=h28bafc1_2
  - aws-c-sdkutils=0.1.19=hc418e29_5
  - aws-checksums=0.1.20=hc418e29_2
  - aws-crt-cpp=0.28.5=h80afcb3_2
  - aws-sdk-cpp=1.11.407=h747b5ae_3
  - azure-core-cpp=1.14.0=h5cfcd09_0
  - azure-identity-cpp=1.10.0=h113e628_0
  - azure-storage-blobs-cpp=12.13.0=h3cf044e_1
  - azure-storage-common-cpp=12.8.0=h736e048_1
  - azure-storage-files-datalake-cpp=12.12.0=ha633028_1
  - beautifulsoup4=4.12.3=pyha770c72_0
  - bleach=6.1.0=pyhd8ed1ab_0
  - blosc=1.21.6=hef167b5_0
  - bokeh=3.6.0=pyhd8ed1ab_0
  - branca=0.7.2=pyhd8ed1ab_0
  - brotli=1.1.0=hb9d3cd8_2
  - brotli-bin=1.1.0=hb9d3cd8_2
  - brotli-python=1.1.0=py312h2ec8cdc_2
  - brunsli=0.1=h9c3ff4c_0
  - bzip2=1.0.8=h4bc722e_7
  - c-ares=1.34.2=heb4867d_0
  - c-blosc2=2.15.1=hc57e6cf_0
  - ca-certificates=2024.8.30=hbcca054_0
  - cached-property=1.5.2=hd8ed1ab_1
  - cached_property=1.5.2=pyha770c72_1
  - cachetools=5.5.0=pyhd8ed1ab_0
  - certifi=2024.8.30=pyhd8ed1ab_0
  - cffi=1.17.1=py312h06ac9bb_0
  - charls=2.4.2=h59595ed_0
  - charset-normalizer=3.4.0=pyhd8ed1ab_0
  - click=8.1.7=unix_pyh707e725_0
  - cloudpickle=3.1.0=pyhd8ed1ab_1
  - colorama=0.4.6=pyhd8ed1ab_0
  - colorcet=3.1.0=pyhd8ed1ab_0
  - contourpy=1.3.0=py312h68727a3_2
  - cucim=24.10.00=cuda12_py312_241009_gf08280c_0
  - cuda-cccl_linux-64=12.5.39=ha770c72_0
  - cuda-crt-dev_linux-64=12.5.82=ha770c72_0
  - cuda-crt-tools=12.5.82=ha770c72_0
  - cuda-cudart=12.5.82=he02047a_0
  - cuda-cudart-dev=12.5.82=he02047a_0
  - cuda-cudart-dev_linux-64=12.5.82=h85509e4_0
  - cuda-cudart-static=12.5.82=he02047a_0
  - cuda-cudart-static_linux-64=12.5.82=h85509e4_0
  - cuda-cudart_linux-64=12.5.82=h85509e4_0
  - cuda-nvcc-dev_linux-64=12.5.82=ha770c72_0
  - cuda-nvcc-impl=12.5.82=hd3aeb46_0
  - cuda-nvcc-tools=12.5.82=hd3aeb46_0
  - cuda-nvrtc=12.5.82=he02047a_0
  - cuda-nvvm-dev_linux-64=12.5.82=ha770c72_0
  - cuda-nvvm-impl=12.5.82=h59595ed_0
  - cuda-nvvm-tools=12.5.82=h59595ed_0
  - cuda-profiler-api=12.5.39=ha770c72_0
  - cuda-python=12.6.0=py312he9d8a76_1
  - cuda-version=12.5=hd4f0392_3
  - cudf=24.10.01=cuda12_py312_241009_g7b0adfa253_0
  - cudf_kafka=24.10.01=cuda12_py312_241009_g7b0adfa253_0
  - cugraph=24.10.00=cuda12_py312_241009_g6f2510afa_0
  - cuml=24.10.00=cuda12_py312_241009_gba7e3ab9c_0
  - cuproj=24.10.00=cuda12_py312_241009_g73184efb_0
  - cupy=13.3.0=py312h7d319b9_2
  - cupy-core=13.3.0=py312h1acd1a8_2
  - cuspatial=24.10.00=cuda12_py312_241009_g73184efb_0
  - custreamz=24.10.01=cuda12_py312_241009_g7b0adfa253_0
  - cuvs=24.10.00=cuda12_py312_241009_g7de3a05_0
  - cuxfilter=24.10.00=cuda12_py312_241009_g8c10c79_0
  - cycler=0.12.1=pyhd8ed1ab_0
  - cyrus-sasl=2.1.27=h54b06d7_7
  - cytoolz=1.0.0=py312h66e93f0_1
  - dask=2024.9.0=pyhd8ed1ab_0
  - dask-core=2024.9.0=pyhd8ed1ab_0
  - dask-cuda=24.10.00=py312_241009_g4e45758_0
  - dask-cudf=24.10.01=cuda12_py312_241009_g7b0adfa253_0
  - dask-expr=1.1.14=pyhd8ed1ab_0
  - datashader=0.16.3=pyhd8ed1ab_0
  - dav1d=1.2.1=hd590300_0
  - defusedxml=0.7.1=pyhd8ed1ab_0
  - distributed=2024.9.0=pyhd8ed1ab_0
  - distributed-ucxx=0.40.00=py3.12_241009_g152901c_0
  - dlpack=0.8=h59595ed_3
  - entrypoints=0.4=pyhd8ed1ab_0
  - exceptiongroup=1.2.2=pyhd8ed1ab_0
  - fasteners=0.17.3=pyhd8ed1ab_0
  - fastrlock=0.8.2=py312h30efb56_2
  - fmt=11.0.2=h434a139_0
  - folium=0.18.0=pyhd8ed1ab_0
  - fonttools=4.54.1=py312h178313f_1
  - fqdn=1.5.1=pyhd8ed1ab_0
  - freetype=2.12.1=h267a509_2
  - freexl=2.0.0=h743c826_0
  - frozenlist=1.4.1=py312h66e93f0_1
  - fsspec=2024.10.0=pyhff2d567_0
  - geopandas=1.0.1=pyhd8ed1ab_1
  - geopandas-base=1.0.1=pyha770c72_1
  - geos=3.13.0=h5888daf_0
  - geotiff=1.7.3=h77b800c_3
  - gflags=2.2.2=h5888daf_1005
  - giflib=5.2.2=hd590300_0
  - glog=0.7.1=hbabe93e_0
  - h2=4.1.0=pyhd8ed1ab_0
  - holoviews=1.19.1=pyhd8ed1ab_0
  - hpack=4.0.0=pyh9f0ad1d_0
  - hyperframe=6.0.1=pyhd8ed1ab_0
  - icu=75.1=he02047a_0
  - idna=3.10=pyhd8ed1ab_0
  - imagecodecs=2024.9.22=py312hf6703b6_0
  - imageio=2.36.0=pyh12aca89_1
  - importlib-metadata=8.5.0=pyha770c72_0
  - importlib_metadata=8.5.0=hd8ed1ab_0
  - importlib_resources=6.4.5=pyhd8ed1ab_0
  - isoduration=20.11.0=pyhd8ed1ab_0
  - jinja2=3.1.4=pyhd8ed1ab_0
  - joblib=1.4.2=pyhd8ed1ab_0
  - json-c=0.18=h6688a6e_0
  - jsonpointer=3.0.0=py312h7900ff3_1
  - jsonschema=4.23.0=pyhd8ed1ab_0
  - jsonschema-specifications=2024.10.1=pyhd8ed1ab_0
  - jsonschema-with-format-nongpl=4.23.0=hd8ed1ab_0
  - jupyter-server-proxy=4.4.0=pyhd8ed1ab_0
  - jupyter_client=8.6.3=pyhd8ed1ab_0
  - jupyter_core=5.7.2=pyh31011fe_1
  - jupyter_events=0.10.0=pyhd8ed1ab_0
  - jupyter_server=2.14.2=pyhd8ed1ab_0
  - jupyter_server_terminals=0.5.3=pyhd8ed1ab_0
  - jupyterlab_pygments=0.3.0=pyhd8ed1ab_1
  - jxrlib=1.1=hd590300_3
  - keyutils=1.6.1=h166bdaf_0
  - kiwisolver=1.4.7=py312h68727a3_0
  - krb5=1.21.3=h659f571_0
  - kvikio=24.10.00=cuda12_py312_241009_g85a88a2_0
  - lazy-loader=0.4=pyhd8ed1ab_1
  - lazy_loader=0.4=pyhd8ed1ab_1
  - lcms2=2.16=hb7c19ff_0
  - ld_impl_linux-64=2.43=h712a8e2_2
  - lerc=4.0.0=h27087fc_0
  - libabseil=20240722.0=cxx17_h5888daf_1
  - libaec=1.1.3=h59595ed_0
  - libarchive=3.7.4=hfca40fe_0
  - libarrow=17.0.0=hdb9dd6d_23_cpu
  - libarrow-acero=17.0.0=h5888daf_23_cpu
  - libarrow-dataset=17.0.0=h5888daf_23_cpu
  - libarrow-substrait=17.0.0=he882d9a_23_cpu
  - libavif16=1.1.1=h104a339_1
  - libblas=3.9.0=25_linux64_openblas
  - libbrotlicommon=1.1.0=hb9d3cd8_2
  - libbrotlidec=1.1.0=hb9d3cd8_2
  - libbrotlienc=1.1.0=hb9d3cd8_2
  - libcblas=3.9.0=25_linux64_openblas
  - libcrc32c=1.1.2=h9c3ff4c_0
  - libcublas=12.5.3.2=he02047a_0
  - libcublas-dev=12.5.3.2=he02047a_0
  - libcucim=24.10.00=cuda12_241009_gf08280c_0
  - libcudf=24.10.01=cuda12_241009_g7b0adfa253_0
  - libcudf_kafka=24.10.01=cuda12_241009_g7b0adfa253_0
  - libcufft=11.2.3.61=he02047a_0
  - libcufile=1.10.1.7=he02047a_0
  - libcufile-dev=1.10.1.7=he02047a_0
  - libcugraph=24.10.00=cuda12_241009_g6f2510afa_0
  - libcugraph_etl=24.10.00=cuda12_241009_g6f2510afa_0
  - libcugraphops=24.10.00=cuda12_241009_g7057cc73_0
  - libcuml=24.10.00=cuda12_241009_gba7e3ab9c_0
  - libcumlprims=24.10.00=cuda12_241009_g0848871_0
  - libcurand=10.3.6.82=he02047a_0
  - libcurand-dev=10.3.6.82=he02047a_0
  - libcurl=8.10.1=hbbe4b11_0
  - libcusolver=11.6.3.83=he02047a_0
  - libcusolver-dev=11.6.3.83=he02047a_0
  - libcusparse=12.5.1.3=he02047a_0
  - libcusparse-dev=12.5.1.3=he02047a_0
  - libcuspatial=24.10.00=cuda12_241009_g73184efb_0
  - libcuvs=24.10.00=cuda12_241009_g7de3a05_0
  - libdeflate=1.22=hb9d3cd8_0
  - libedit=3.1.20191231=he28a2e2_2
  - libev=4.33=hd590300_2
  - libevent=2.1.12=hf998b51_1
  - libexpat=2.6.3=h5888daf_0
  - libffi=3.4.2=h7f98852_5
  - libgcc=14.2.0=h77fa898_1
  - libgcc-ng=14.2.0=h69a702a_1
  - libgdal-core=3.9.2=hd5b9bfb_7
  - libgfortran=14.2.0=h69a702a_1
  - libgfortran-ng=14.2.0=h69a702a_1
  - libgfortran5=14.2.0=hd5240d6_1
  - libgomp=14.2.0=h77fa898_1
  - libgoogle-cloud=2.30.0=h438788a_0
  - libgoogle-cloud-storage=2.30.0=h0121fbd_0
  - libgrpc=1.65.5=hf5c653b_0
  - libhwy=1.1.0=h00ab1b0_0
  - libiconv=1.17=hd590300_2
  - libjpeg-turbo=3.0.0=hd590300_1
  - libjxl=0.11.0=hdb8da77_2
  - libkml=1.3.0=hf539b9f_1021
  - libkvikio=24.10.00=cuda12_241009_g85a88a2_0
  - liblapack=3.9.0=25_linux64_openblas
  - libllvm14=14.0.6=hcd5def8_4
  - libnghttp2=1.64.0=h161d5f1_0
  - libnl=3.10.0=h4bc722e_0
  - libnsl=2.0.1=hd590300_0
  - libntlm=1.4=h7f98852_1002
  - libnvjitlink=12.5.82=he02047a_0
  - libnvjpeg=12.3.2.81=he02047a_0
  - libopenblas=0.3.28=pthreads_h94d23a6_0
  - libparquet=17.0.0=h6bd9018_23_cpu
  - libpng=1.6.44=hadc24fc_0
  - libprotobuf=5.27.5=h5b01275_2
  - libraft=24.10.00=cuda12_241009_g397042a0_0
  - libraft-headers=24.10.00=cuda12_241009_g397042a0_0
  - libraft-headers-only=24.10.00=cuda12_241009_g397042a0_0
  - librdkafka=2.5.3=h95ba008_0
  - libre2-11=2024.07.02=hbbce691_1
  - librmm=24.10.00=cuda12_241009_g3223f841_0
  - librttopo=1.1.0=h97f6797_17
  - libsodium=1.0.20=h4ab18f5_0
  - libspatialite=5.1.0=h1b4f908_11
  - libsqlite=3.47.0=hadc24fc_0
  - libssh2=1.11.0=h0841786_0
  - libstdcxx=14.2.0=hc0a3c3a_1
  - libstdcxx-ng=14.2.0=h4852527_1
  - libthrift=0.21.0=h0e7cc3e_0
  - libtiff=4.7.0=he137b08_1
  - libucxx=0.40.00=cuda12_241009_g152901c_0
  - libutf8proc=2.8.0=h166bdaf_0
  - libuuid=2.38.1=h0b41bf4_0
  - libuv=1.49.2=hb9d3cd8_0
  - libwebp-base=1.4.0=hd590300_0
  - libxcb=1.17.0=h8a09558_0
  - libxcrypt=4.4.36=hd590300_1
  - libxgboost=2.1.1=rapidsai_h01f03eb_5
  - libxml2=2.12.7=he7c6b58_4
  - libzlib=1.3.1=hb9d3cd8_2
  - libzopfli=1.0.3=h9c3ff4c_0
  - linkify-it-py=2.0.3=pyhd8ed1ab_0
  - llvmlite=0.43.0=py312h374181b_1
  - locket=1.0.0=pyhd8ed1ab_0
  - lz4=4.3.3=py312hb3f7f12_1
  - lz4-c=1.9.4=hcb278e6_0
  - lzo=2.10=hd590300_1001
  - mapclassify=2.8.1=pyhd8ed1ab_0
  - markdown=3.6=pyhd8ed1ab_0
  - markdown-it-py=3.0.0=pyhd8ed1ab_0
  - markupsafe=3.0.2=py312h178313f_0
  - matplotlib-base=3.9.2=py312hd3ec401_1
  - mdit-py-plugins=0.4.2=pyhd8ed1ab_0
  - mdurl=0.1.2=pyhd8ed1ab_0
  - minizip=4.0.7=h401b404_0
  - mistune=3.0.2=pyhd8ed1ab_0
  - msgpack-python=1.1.0=py312h68727a3_0
  - multidict=6.1.0=py312h178313f_1
  - multipledispatch=0.6.0=pyhd8ed1ab_1
  - munkres=1.1.4=pyh9f0ad1d_0
  - nbclient=0.10.0=pyhd8ed1ab_0
  - nbconvert-core=7.16.4=pyhd8ed1ab_1
  - nbformat=5.10.4=pyhd8ed1ab_0
  - nccl=2.23.4.1=h52f6c39_1
  - ncurses=6.5=he02047a_1
  - networkx=3.4.2=pyhd8ed1ab_0
  - nodejs=22.9.0=hf235a45_0
  - numba=0.60.0=py312h83e6fd3_0
  - numcodecs=0.13.1=py312hf9745cd_0
  - numpy=2.0.2=py312h58c1407_0
  - nvcomp=4.0.1=hbc370b7_0
  - nvtx=0.2.10=py312h66e93f0_2
  - nx-cugraph=24.10.00=py312_241009_g6f2510afa_0
  - openjpeg=2.5.2=h488ebb8_0
  - openssl=3.3.2=hb9d3cd8_0
  - orc=2.0.2=h690cf93_1
  - overrides=7.7.0=pyhd8ed1ab_0
  - packaging=24.1=pyhd8ed1ab_0
  - pandas=2.2.2=py312h1d6d2e6_1
  - pandocfilters=1.5.0=pyhd8ed1ab_0
  - panel=1.5.2=pyhd8ed1ab_0
  - param=2.1.1=pyhff2d567_0
  - partd=1.4.2=pyhd8ed1ab_0
  - pcre2=10.44=hba22ea6_2
  - pillow=11.0.0=py312h7b63e92_0
  - pip=24.2=pyh8b19718_1
  - pkgutil-resolve-name=1.3.10=pyhd8ed1ab_1
  - platformdirs=4.3.6=pyhd8ed1ab_0
  - proj=9.5.0=h12925eb_0
  - prometheus_client=0.21.0=pyhd8ed1ab_0
  - propcache=0.2.0=py312h66e93f0_2
  - psutil=6.0.0=py312h66e93f0_2
  - pthread-stubs=0.4=hb9d3cd8_1002
  - ptyprocess=0.7.0=pyhd3deb0d_0
  - py-xgboost=2.1.1=rapidsai_pyh53d8b89_5
  - pyarrow=17.0.0=py312h9cebb41_1
  - pyarrow-core=17.0.0=py312h9cafe31_1_cpu
  - pycparser=2.22=pyhd8ed1ab_0
  - pyct=0.5.0=pyhd8ed1ab_0
  - pygments=2.18.0=pyhd8ed1ab_0
  - pylibcudf=24.10.01=cuda12_py312_241009_g7b0adfa253_0
  - pylibcugraph=24.10.00=cuda12_py312_241009_g6f2510afa_0
  - pylibraft=24.10.00=cuda12_py312_241009_g397042a0_0
  - pynvjitlink=0.3.0=py312hd269673_0
  - pynvml=11.4.1=pyhd8ed1ab_0
  - pyogrio=0.10.0=py312he8b4914_0
  - pyparsing=3.2.0=pyhd8ed1ab_1
  - pyproj=3.7.0=py312he630544_0
  - pysocks=1.7.1=pyha2e5f31_6
  - python=3.12.7=hc5c86c4_0_cpython
  - python-confluent-kafka=2.5.3=py312h66e93f0_0
  - python-dateutil=2.9.0=pyhd8ed1ab_0
  - python-fastjsonschema=2.20.0=pyhd8ed1ab_0
  - python-json-logger=2.0.7=pyhd8ed1ab_0
  - python-tzdata=2024.2=pyhd8ed1ab_0
  - python_abi=3.12=5_cp312
  - pytz=2024.2=pyhd8ed1ab_0
  - pyviz_comms=3.0.3=pyhd8ed1ab_0
  - pywavelets=1.7.0=py312hc0a28a1_2
  - pyyaml=6.0.2=py312h66e93f0_1
  - pyzmq=26.2.0=py312hbf22597_3
  - qhull=2020.2=h434a139_5
  - raft-dask=24.10.00=cuda12_py312_241009_g397042a0_0
  - rapids=24.10.00=cuda12_py312_241009_g19a0c5a_0
  - rapids-dask-dependency=24.10.00=py_0
  - rapids-xgboost=24.10.00=cuda12_py312_241009_g19a0c5a_0
  - rav1e=0.6.6=he8a937b_2
  - rdma-core=54.0=h5888daf_0
  - re2=2024.07.02=h77b4e00_1
  - readline=8.2=h8228510_1
  - referencing=0.35.1=pyhd8ed1ab_0
  - requests=2.32.3=pyhd8ed1ab_0
  - rfc3339-validator=0.1.4=pyhd8ed1ab_0
  - rfc3986-validator=0.1.1=pyh9f0ad1d_0
  - rich=13.9.3=pyhd8ed1ab_0
  - rmm=24.10.00=cuda12_py312_241009_g3223f841_0
  - rpds-py=0.20.0=py312h12e396e_1
  - s2n=1.5.5=h3931f03_0
  - scikit-image=0.24.0=py312h1df14c2_2
  - scikit-learn=1.5.2=py312h7a48858_1
  - scipy=1.14.1=py312h62794b6_1
  - send2trash=1.8.3=pyh0d859eb_0
  - setuptools=75.1.0=pyhd8ed1ab_0
  - shapely=2.0.6=py312h391bc85_2
  - simpervisor=1.0.0=pyhd8ed1ab_0
  - six=1.16.0=pyh6c4a22f_0
  - snappy=1.2.1=ha2e4443_0
  - sniffio=1.3.1=pyhd8ed1ab_0
  - sortedcontainers=2.4.0=pyhd8ed1ab_0
  - soupsieve=2.5=pyhd8ed1ab_1
  - spdlog=1.14.1=hed91bc2_1
  - sqlite=3.47.0=h9eae976_0
  - streamz=0.6.4=pyh6c4a22f_0
  - svt-av1=2.2.1=h5888daf_0
  - tblib=3.0.0=pyhd8ed1ab_0
  - terminado=0.18.1=pyh0d859eb_0
  - threadpoolctl=3.5.0=pyhc1e730c_0
  - tifffile=2024.9.20=pyhd8ed1ab_0
  - tinycss2=1.3.0=pyhd8ed1ab_0
  - tk=8.6.13=noxft_h4845f30_101
  - toolz=1.0.0=pyhd8ed1ab_0
  - tornado=6.4.1=py312h66e93f0_1
  - tqdm=4.66.5=pyhd8ed1ab_0
  - traitlets=5.14.3=pyhd8ed1ab_0
  - treelite=4.3.0=py312h01abfbf_0
  - types-python-dateutil=2.9.0.20241003=pyhff2d567_0
  - typing-extensions=4.12.2=hd8ed1ab_0
  - typing_extensions=4.12.2=pyha770c72_0
  - typing_utils=0.1.0=pyhd8ed1ab_0
  - tzdata=2024b=hc8b5060_0
  - uc-micro-py=1.0.3=pyhd8ed1ab_0
  - ucx=1.17.0=h05e919c_3
  - ucx-proc=1.0.0=gpu
  - ucx-py=0.40.00=py312_241009_g773cd1e_0
  - ucxx=0.40.00=cuda12_py3.12_241009_g152901c_0
  - unicodedata2=15.1.0=py312h98912ed_0
  - uri-template=1.3.0=pyhd8ed1ab_0
  - uriparser=0.9.8=hac33072_0
  - urllib3=2.2.3=pyhd8ed1ab_0
  - webcolors=24.8.0=pyhd8ed1ab_0
  - webencodings=0.5.1=pyhd8ed1ab_2
  - websocket-client=1.8.0=pyhd8ed1ab_0
  - wheel=0.44.0=pyhd8ed1ab_0
  - xarray=2024.9.0=pyhd8ed1ab_1
  - xerces-c=3.2.5=h988505b_2
  - xgboost=2.1.1=rapidsai_pyh9bdd636_5
  - xorg-libxau=1.0.11=hb9d3cd8_1
  - xorg-libxdmcp=1.1.5=hb9d3cd8_0
  - xyzservices=2024.9.0=pyhd8ed1ab_0
  - xz=5.2.6=h166bdaf_0
  - yaml=0.2.5=h7f98852_2
  - yarl=1.15.5=py312h66e93f0_0
  - zarr=2.18.3=pyhd8ed1ab_0
  - zeromq=4.3.5=h3b0a872_6
  - zfp=1.0.1=h5888daf_2
  - zict=3.0.0=pyhd8ed1ab_0
  - zipp=3.20.2=pyhd8ed1ab_0
  - zlib=1.3.1=hb9d3cd8_2
  - zlib-ng=2.2.2=h5888daf_0
  - zstandard=0.23.0=py312hef9b889_1
  - zstd=1.5.6=ha6fb4c9_0
prefix: /lustre/mkristensen/miniconda3/envs/rapids-24.10

This is on a DGX-1, @fstrug what system are you running on?

@fstrug fstrug mentioned this issue Oct 23, 2024
Open
@fstrug
Copy link

fstrug commented Oct 23, 2024

@madsbk I will contact our sys admins since I'm not sure and post into the new issue.

@fstrug
Copy link

fstrug commented Oct 23, 2024 •
edited
Loading

In the build case expect we need to pass more arguments to cmake like

cmake ..
-DCMAKE_PREFIX_PATH="${CONDA_PREFIX}"
-DCMAKE_INSTALL_PREFIX:PATH="${CONDA_PREFIX}"
...
;

Though we do have build.sh, which may help with that. So one could do the following to build and install the C++ and Python packages into your Conda env (the install location can also be customized)

Unfortunately neither of these resolved the cuFile not found error. Passing the arguments to cmake gives the same output.

Building from source seemed to mostly work. Our systems drivers are version 12.2, so I've been trying to pin cuda-version=12.2. Perhaps there are some backwards compatibility issues, since we have encountered this ourselves when trying to set-up images with cuda-version=12.6 and using nvcc.

mamba env create --name kvikio-dev --file conda/environments/all_cuda-125_arch-x86_64.yaml
./build.sh libkvikio kvikio

it was able to find cuFile and for the test I get:

(kvikio-dev-clone) [fstrug@jupyter-fstrug kvikio]$ ./build.sh libkvikio kvikio                                                                                                                                                                                                                                 
Executing cmake for libkvikio...                                                                                                                                                                                                                                                                               
-- The CXX compiler identification is GNU 11.4.0                                                                                                                                                                                                                                                               
-- Detecting CXX compiler ABI info                                                                                                                                                                                                                                                                             
-- Detecting CXX compiler ABI info - done                                                                                                                                                                                                                                                                      
-- Check for working CXX compiler: /home/fstrug/.conda/envs/kvikio-dev-clone/bin/x86_64-conda-linux-gnu-c++ - skipped                                                                                                                                                                                          
-- Detecting CXX compile features                                                                                                                                                                                                                                                                              
-- Detecting CXX compile features - done                                                                                                                                                                                                                                                                       
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD                                                                                                                                                                                                                                                                     
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed                                                                                                                                                                                                                                                            
-- Looking for pthread_create in pthreads                                                                                                                                                                                                                                                                      
-- Looking for pthread_create in pthreads - not found                                                                                                                                                                                                                                                          
-- Looking for pthread_create in pthread                                                                                                                                                                                                                                                                       
-- Looking for pthread_create in pthread - found                                                                                                                                                                                                                                                               
-- Found Threads: TRUE                                                                                                                                                                                                                                                                                         
-- CPM: Using local package [email protected]                                                                                                                                                                                                                                                                        
nvcc warning : incompatible redefinition for option 'compiler-bindir', the last value of this option was used                                                                                                                                                                                                  
-- Found CUDAToolkit: /home/fstrug/.conda/envs/kvikio-dev-clone/targets/x86_64-linux/include (found version "12.5.82")                                                                                                                                                                                         
nvcc warning : incompatible redefinition for option 'compiler-bindir', the last value of this option was used                                                                                                                                                                                                  
-- Found cuFile: /home/fstrug/.conda/envs/kvikio-dev-clone/lib/libcufile.so                                                                                                                                                                                                                                    
-- Found cuFile Batch API: TRUE                                                                                                                                                                                                                                                                                
-- Found cuFile Stream API: TRUE                                                                                                                                                                                                                                                                               
-- CPM: Adding package [email protected] (v3.1.0)                                                                                                                                                                                                                                                                    
CMake Warning (dev) at /home/fstrug/.conda/envs/kvikio-dev-clone/share/cmake-3.30/Modules/FetchContent.cmake:1953 (message):                                                                                                                                                                                   
  Calling FetchContent_Populate(nvtx3) is deprecated, call                                                                                                                                                                                                                                                     
  FetchContent_MakeAvailable(nvtx3) instead.  Policy CMP0169 can be set to                                                                                                                                                                                                                                     
  OLD to allow FetchContent_Populate(nvtx3) to be called directly for now,                                                                                                                                                                                                                                     
  but the ability to call it with declared details will be removed completely                                                                                                                                                                                                                                  
  in a future version.                                                                                                                                                                                                                                                                                         
Call Stack (most recent call first):                                                                                                                                                                                                                                                                           
  build/cmake/CPM_0.40.0.cmake:1074 (FetchContent_Populate)                                                                                                                                                                                                                                                    
  build/cmake/CPM_0.40.0.cmake:868 (cpm_fetch_package)                                                                                                                                                                                                                                                         
  build/cmake/CPM_0.40.0.cmake:306 (CPMAddPackage)                                                                                                                                                                                                                                                             
  build/_deps/rapids-cmake-src/rapids-cmake/cpm/find.cmake:189 (CPMFindPackage)                                                                                                                                                                                                                                
  build/_deps/rapids-cmake-src/rapids-cmake/cpm/nvtx3.cmake:75 (rapids_cpm_find)                                                                                                                                                                                                                               
  cmake/thirdparty/get_nvtx.cmake:21 (rapids_cpm_nvtx3)                                                                                                                                                                                                                                                        
  cmake/thirdparty/get_nvtx.cmake:25 (find_and_configure_nvtx)                                                                                                                                                                                                                                                 
  CMakeLists.txt:92 (include)                                                                                                                                                                                                                                                                                  
This warning is for project developers.  Use -Wno-dev to suppress it.                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                                                               
-- CPM: Adding package [email protected] (097aa718f25d44315cadb80b407144ad455ee4f9)                                                                                                                                                                                                                         
CMake Warning (dev) at /home/fstrug/.conda/envs/kvikio-dev-clone/share/cmake-3.30/Modules/FetchContent.cmake:1953 (message):                                                                                                                                                                                   
  Calling FetchContent_Populate(bs_thread_pool) is deprecated, call                                                                                                                                                                                                                                            
  FetchContent_MakeAvailable(bs_thread_pool) instead.  Policy CMP0169 can be                                                                                                                                                                                                                                   
  set to OLD to allow FetchContent_Populate(bs_thread_pool) to be called                                                                                                                                                                                                                                       
  directly for now, but the ability to call it with declared details will be                                                                                                                                                                                                                                   
  removed completely in a future version.                                                                                                                                                                                                                                                                      
Call Stack (most recent call first):                                                                                                                                                                                                                                                                           
  build/cmake/CPM_0.40.0.cmake:1074 (FetchContent_Populate)                                                                                                                                                                                                                                                    
  build/cmake/CPM_0.40.0.cmake:868 (cpm_fetch_package)
  build/cmake/CPM_0.40.0.cmake:306 (CPMAddPackage)
  build/_deps/rapids-cmake-src/rapids-cmake/cpm/find.cmake:189 (CPMFindPackage)
  build/_deps/rapids-cmake-src/rapids-cmake/cpm/bs_thread_pool.cmake:74 (rapids_cpm_find)
  cmake/thirdparty/get_thread_pool.cmake:21 (rapids_cpm_bs_thread_pool)
  cmake/thirdparty/get_thread_pool.cmake:25 (find_and_configure_thread_pool)
  CMakeLists.txt:95 (include)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- CPM: Adding package [email protected] (v1.13.0)
CMake Warning (dev) at /home/fstrug/.conda/envs/kvikio-dev-clone/share/cmake-3.30/Modules/FetchContent.cmake:1953 (message):
  Calling FetchContent_Populate(GTest) is deprecated, call
  FetchContent_MakeAvailable(GTest) instead.  Policy CMP0169 can be set to
  OLD to allow FetchContent_Populate(GTest) to be called directly for now,
  but the ability to call it with declared details will be removed completely
  in a future version.
Call Stack (most recent call first):
  build/cmake/CPM_0.40.0.cmake:1074 (FetchContent_Populate)
  build/cmake/CPM_0.40.0.cmake:868 (cpm_fetch_package)
  build/cmake/CPM_0.40.0.cmake:298 (CPMAddPackage)
  build/_deps/rapids-cmake-src/rapids-cmake/cpm/find.cmake:189 (CPMFindPackage)
  build/_deps/rapids-cmake-src/rapids-cmake/cpm/gtest.cmake:79 (rapids_cpm_find)
  cmake/thirdparty/get_gtest.cmake:20 (rapids_cpm_gtest)
  cmake/thirdparty/get_gtest.cmake:24 (find_and_configure_gtest)
  CMakeLists.txt:161 (include)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- The C compiler identification is GNU 11.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /home/fstrug/.conda/envs/kvikio-dev-clone/bin/x86_64-conda-linux-gnu-cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Found Python: /home/fstrug/.conda/envs/kvikio-dev-clone/bin/python3.12 (found version "3.12.7") found components: Interpreter
nvcc warning : incompatible redefinition for option 'compiler-bindir', the last value of this option was used
nvcc warning : incompatible redefinition for option 'compiler-bindir', the last value of this option was used
-- Configuring done (11.0s)
-- Generating done (0.2s)
-- Build files have been written to: /home/fstrug/kvikio/cpp/build
building libkvikio...
[ 13%] Building CXX object examples/CMakeFiles/BASIC_IO_EXAMPLE.dir/basic_io.cpp.o
[ 13%] Building CXX object examples/CMakeFiles/BASIC_NO_CUDA_EXAMPLE.dir/basic_no_cuda.cpp.o
[ 20%] Building CXX object _deps/gtest-build/googletest/CMakeFiles/gtest.dir/src/gtest-all.cc.o
[ 26%] Linking CXX executable BASIC_NO_CUDA_EXAMPLE
[ 26%] Built target BASIC_NO_CUDA_EXAMPLE
[ 33%] Linking CXX executable BASIC_IO_EXAMPLE
[ 33%] Built target BASIC_IO_EXAMPLE
[ 40%] Linking CXX static library ../../../lib/libgtest.a
[ 40%] Built target gtest
[ 46%] Building CXX object _deps/gtest-build/googletest/CMakeFiles/gtest_main.dir/src/gtest_main.cc.o
[ 53%] Building CXX object _deps/gtest-build/googlemock/CMakeFiles/gmock.dir/src/gmock-all.cc.o
[ 60%] Linking CXX static library ../../../lib/libgtest_main.a
[ 60%] Built target gtest_main
[ 66%] Linking CXX static library ../../../lib/libgmock.a
[ 66%] Built target gmock
[ 73%] Building CXX object _deps/gtest-build/googlemock/CMakeFiles/gmock_main.dir/src/gmock_main.cc.o
[ 86%] Building CXX object tests/CMakeFiles/cpp_tests.dir/test_basic_io.cpp.o
[ 86%] Building CXX object tests/CMakeFiles/cpp_tests.dir/main.cpp.o
[ 93%] Linking CXX static library ../../../lib/libgmock_main.a
[ 93%] Built target gmock_main
[100%] Linking CXX executable ../gtests/cpp_tests
[100%] Built target cpp_tests
installing libkvikio...
[ 13%] Built target BASIC_IO_EXAMPLE
[ 26%] Built target BASIC_NO_CUDA_EXAMPLE
[ 40%] Built target gtest
[ 53%] Built target gmock
[ 66%] Built target gmock_main
[ 80%] Built target gtest_main
[100%] Built target cpp_tests
Install the project...
-- Install configuration: "Release"
-- Up-to-date: /home/fstrug/.conda/envs/kvikio-dev-clone/include
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/nvtx3
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/nvtx3/nvToolsExtCudaRt.h
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/nvtx3/nvtxDetail
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/nvtx3/nvtxDetail/nvtxImplOpenCL_v3.h
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/nvtx3/nvtxDetail/nvtxTypes.h
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/nvtx3/nvtxDetail/nvtxInitDecls.h
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/nvtx3/nvtxDetail/nvtxImplSync_v3.h
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/nvtx3/nvtxDetail/nvtxInitDefs.h
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/nvtx3/nvtxDetail/nvtxImplCuda_v3.h
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/nvtx3/nvtxDetail/nvtxInit.h
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/nvtx3/nvtxDetail/nvtxImpl.h
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/nvtx3/nvtxDetail/nvtxLinkOnce.h
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/nvtx3/nvtxDetail/nvtxImplCore.h
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/nvtx3/nvtxDetail/nvtxImplCudaRt_v3.h
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/nvtx3/nvToolsExtCuda.h
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/nvtx3/nvtx3.hpp
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/nvtx3/nvToolsExtOpenCL.h
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/nvtx3/nvToolsExt.h
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/nvtx3/nvToolsExtSync.h
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/lib/cmake/nvtx3/nvtx3-targets.cmake
-- Up-to-date: /home/fstrug/.conda/envs/kvikio-dev-clone/lib/cmake/nvtx3
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/lib/cmake/nvtx3/nvtx3-config-version.cmake
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/lib/cmake/nvtx3/nvtx3-config.cmake
-- Up-to-date: /home/fstrug/.conda/envs/kvikio-dev-clone/include
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/BS_thread_pool_utils.hpp
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/BS_thread_pool.hpp
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/lib/cmake/bs_thread_pool/bs_thread_pool-targets.cmake
-- Up-to-date: /home/fstrug/.conda/envs/kvikio-dev-clone/lib/cmake/bs_thread_pool
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/lib/cmake/bs_thread_pool/bs_thread_pool-config-version.cmake
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/lib/cmake/bs_thread_pool/bs_thread_pool-config.cmake
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/kvikio
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/kvikio/stream.hpp
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/kvikio/driver.hpp
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/kvikio/cufile_config.hpp
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/kvikio/parallel_operation.hpp
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/kvikio/utils.hpp
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/kvikio/posix_io.hpp
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/kvikio/shim
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/kvikio/shim/utils.hpp
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/kvikio/shim/cufile.hpp
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/kvikio/shim/libcurl.hpp
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/kvikio/shim/cuda_h_wrapper.hpp
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/kvikio/shim/cufile_h_wrapper.hpp
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/kvikio/shim/cuda.hpp
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/kvikio/buffer.hpp
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/kvikio/bounce_buffer.hpp
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/kvikio/error.hpp
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/kvikio/batch.hpp
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/kvikio/defaults.hpp
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/kvikio/remote_handle.hpp
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/kvikio/file_handle.hpp
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/include/kvikio/version_config.hpp
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/lib/cmake/kvikio/kvikio-targets.cmake
-- Up-to-date: /home/fstrug/.conda/envs/kvikio-dev-clone/lib/cmake/kvikio
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/lib/cmake/kvikio/kvikio-config.cmake
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/lib/cmake/kvikio/kvikio-config-version.cmake
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/lib/cmake/kvikio/kvikio-dependencies.cmake
-- Installing: /home/fstrug/.conda/envs/kvikio-dev-clone/lib/cmake/kvikio/FindcuFile.cmake
building kvikio...
Using pip 24.2 from /home/fstrug/.conda/envs/kvikio-dev-clone/lib/python3.12/site-packages/pip (python 3.12)
Processing /home/fstrug/kvikio/python/kvikio
  Running command Preparing metadata (pyproject.toml)
  *** scikit-build-core 0.10.7 using CMake 3.26.5 (metadata_wheel)
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: kvikio
  Running command Building wheel for kvikio (pyproject.toml)
  *** scikit-build-core 0.10.7 using CMake 3.26.5 (wheel)
  *** Configuring CMake...
  2024-10-23 15:04:19,867 - scikit_build_core - WARNING - Unsupported CMAKE_ARGS ignored: -DCMAKE_BUILD_TYPE=Release
  2024-10-23 15:04:20,432 - scikit_build_core - WARNING - Unsupported CMAKE_ARGS ignored: -DCMAKE_BUILD_TYPE=Release
  CMake Warning:
    Ignoring empty string ("") provided on the command line.


  loading initial cache file build/cp312-cp312-linux_x86_64/CMakeInit.txt
  -- The CXX compiler identification is GNU 11.4.0
  -- The CUDA compiler identification is NVIDIA 12.5.82
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Check for working CXX compiler: /home/fstrug/.conda/envs/kvikio-dev-clone/bin/x86_64-conda-linux-gnu-c++ - skipped
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  -- Detecting CUDA compiler ABI info
  -- Detecting CUDA compiler ABI info - done
  -- Check for working CUDA compiler: /home/fstrug/.conda/envs/kvikio-dev-clone/bin/nvcc - skipped
  -- Detecting CUDA compile features
  -- Detecting CUDA compile features - done
  -- Project kvikio-python is building for CUDA architectures:
    70-real
    75-real
    80-real
    86-real
    90
  -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
  -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
  -- Looking for pthread_create in pthreads
  -- Looking for pthread_create in pthreads - not found
  -- Looking for pthread_create in pthread
  -- Looking for pthread_create in pthread - found
  -- Found Threads: TRUE
  -- Found cuFile: /home/fstrug/.conda/envs/kvikio-dev-clone/lib/libcufile.so
  -- Found CURL: /home/fstrug/.conda/envs/kvikio-dev-clone/lib/libcurl.so (found version "8.10.1")
  -- Unable to find cublas_v2.h in either "/home/fstrug/.conda/envs/kvikio-dev-clone/targets/x86_64-linux/include" or "/home/fstrug/.conda/envs/kvikio-dev-clone/math_libs/include"
  -- Found CUDAToolkit: /home/fstrug/.conda/envs/kvikio-dev-clone/targets/x86_64-linux/include (found version "12.5.82")
  -- Found nvtx3: /home/fstrug/.conda/envs/kvikio-dev-clone/lib/cmake/nvtx3/nvtx3-config.cmake (found version "3.1.0")
  -- Found bs_thread_pool: /home/fstrug/.conda/envs/kvikio-dev-clone/lib/cmake/bs_thread_pool/bs_thread_pool-config.cmake (found version "4.1.0")
  -- Found KvikIO: /home/fstrug/.conda/envs/kvikio-dev-clone/lib/cmake/kvikio/kvikio-config.cmake (found version "24.12.0")
  -- KvikIO: Found cuFile Batch API: TRUE
  -- KvikIO: Found cuFile Stream API: TRUE
  -- Unable to find cublas_v2.h in either "/home/fstrug/.conda/envs/kvikio-dev-clone/targets/x86_64-linux/include" or "/home/fstrug/.conda/envs/kvikio-dev-clone/math_libs/include"
  -- Found Python: /home/fstrug/.conda/envs/kvikio-dev-clone/bin/python (found version "3.12.7") found components: Interpreter Development.Module
  -- Unable to find cublas_v2.h in either "/home/fstrug/.conda/envs/kvikio-dev-clone/targets/x86_64-linux/include" or "/home/fstrug/.conda/envs/kvikio-dev-clone/math_libs/include"
  -- Found nvcomp: /home/fstrug/.conda/envs/kvikio-dev-clone/lib/cmake/nvcomp (found version 4.0.1.0)
  -- Building remote_handle.pyx (libcurl found)
  -- Configuring done (6.3s)
  -- Generating done (0.1s)
  CMake Warning:
    Manually-specified variables were not used by the project:

      CMAKE_C_COMPILER_AR
      CMAKE_C_COMPILER_RANLIB


  -- Build files have been written to: /home/fstrug/kvikio/python/kvikio/build/cp312-cp312-linux_x86_64
  2024-10-23 15:04:26,931 - scikit_build_core - WARNING - Unsupported CMAKE_ARGS ignored: -DCMAKE_BUILD_TYPE=Release
  *** Building project with Ninja...
  [1/27] Transpiling driver_properties.pyx to driver_properties.cxx
  [2/27] Transpiling future.pyx to future.cxx
  [3/27] Transpiling defaults.pyx to defaults.cxx
  [4/27] Transpiling libnvcomp_ll.pyx to libnvcomp_ll.cxx
  [5/27] Transpiling buffer.pyx to buffer.cxx
  [6/27] Transpiling file_handle.pyx to file_handle.cxx
  [7/27] Transpiling libnvcomp.pyx to libnvcomp.cxx
  [8/27] Transpiling remote_handle.pyx to remote_handle.cxx
  [9/27] Transpiling arr.pyx to arr.cxx
  [10/27] Building CXX object kvikio/_lib/CMakeFiles/driver_properties.dir/driver_properties.cxx.o
  [11/27] Linking CXX shared module kvikio/_lib/driver_properties.cpython-312-x86_64-linux-gnu.so
  [12/27] Building CXX object kvikio/_lib/CMakeFiles/defaults.dir/defaults.cxx.o
  [13/27] Building CXX object kvikio/_lib/CMakeFiles/future.dir/future.cxx.o
  [14/27] Linking CXX shared module kvikio/_lib/defaults.cpython-312-x86_64-linux-gnu.so
  [15/27] Linking CXX shared module kvikio/_lib/future.cpython-312-x86_64-linux-gnu.so
  [16/27] Building CXX object kvikio/_lib/CMakeFiles/libnvcomp_ll.dir/libnvcomp_ll.cxx.o
  [17/27] Linking CXX shared module kvikio/_lib/libnvcomp_ll.cpython-312-x86_64-linux-gnu.so
  [18/27] Building CXX object kvikio/_lib/CMakeFiles/arr.dir/arr.cxx.o
  [19/27] Linking CXX shared module kvikio/_lib/arr.cpython-312-x86_64-linux-gnu.so
  [20/27] Building CXX object kvikio/_lib/CMakeFiles/libnvcomp.dir/libnvcomp.cxx.o
  /home/fstrug/kvikio/python/kvikio/build/cp312-cp312-linux_x86_64/kvikio/_lib/libnvcomp.cxx: In function 'int __pyx_pf_6kvikio_4_lib_9libnvcomp_11_ANSManager___cinit__(__pyx_obj_6kvikio_4_lib_9libnvcomp__ANSManager*, size_t, PyObject*, int)':
  /home/fstrug/kvikio/python/kvikio/build/cp312-cp312-linux_x86_64/kvikio/_lib/libnvcomp.cxx:20326:142: warning: 'nvcomp::ANSManager::ANSManager(size_t, const nvcompBatchedANSOpts_t&, cudaStream_t, int, nvcomp::ChecksumPolicy, nvcomp::BitstreamKind)' is deprecated [-Wdeprecated-declarations]
  20326 |     __pyx_t_1 = new nvcomp::ANSManager(__pyx_v_uncomp_chunk_size, nvcompBatchedANSDefaultOpts, ((cudaStream_t)((void *)0)), __pyx_v_device_id);
        |                                                                                                                                              ^
  In file included from /home/fstrug/.conda/envs/kvikio-dev-clone/include/nvcomp/nvcompManagerFactory.hpp:16,
                   from /home/fstrug/kvikio/python/kvikio/build/cp312-cp312-linux_x86_64/kvikio/_lib/libnvcomp.cxx:1257:
  /home/fstrug/.conda/envs/kvikio-dev-clone/include/nvcomp/ans.hpp:42:18: note: declared here
     42 |   [[deprecated]] ANSManager(
        |                  ^~~~~~~~~~
  /home/fstrug/kvikio/python/kvikio/build/cp312-cp312-linux_x86_64/kvikio/_lib/libnvcomp.cxx: In function 'int __pyx_pf_6kvikio_4_lib_9libnvcomp_15_BitcompManager___cinit__(__pyx_obj_6kvikio_4_lib_9libnvcomp__BitcompManager*, size_t, nvcompType_t, int, PyObject*, int)':
  /home/fstrug/kvikio/python/kvikio/build/cp312-cp312-linux_x86_64/kvikio/_lib/libnvcomp.cxx:20764:128: warning: 'nvcomp::BitcompManager::BitcompManager(size_t, const nvcompBatchedBitcompFormatOpts&, cudaStream_t, int, nvcomp::ChecksumPolicy, nvcomp::BitstreamKind)' is deprecated [-Wdeprecated-declarations]
  20764 |     __pyx_t_4 = new nvcomp::BitcompManager(__pyx_v_uncomp_chunk_size, __pyx_t_3, ((cudaStream_t)((void *)0)), __pyx_v_device_id);
        |                                                                                                                                ^
  In file included from /home/fstrug/.conda/envs/kvikio-dev-clone/include/nvcomp/nvcompManagerFactory.hpp:20,
                   from /home/fstrug/kvikio/python/kvikio/build/cp312-cp312-linux_x86_64/kvikio/_lib/libnvcomp.cxx:1257:
  /home/fstrug/.conda/envs/kvikio-dev-clone/include/nvcomp/bitcomp.hpp:57:18: note: declared here
     57 |   [[deprecated]] BitcompManager(
        |                  ^~~~~~~~~~~~~~
  /home/fstrug/kvikio/python/kvikio/build/cp312-cp312-linux_x86_64/kvikio/_lib/libnvcomp.cxx: In function 'int __pyx_pf_6kvikio_4_lib_9libnvcomp_16_CascadedManager___cinit__(__pyx_obj_6kvikio_4_lib_9libnvcomp__CascadedManager*, PyObject*, PyObject*, int)':
  /home/fstrug/kvikio/python/kvikio/build/cp312-cp312-linux_x86_64/kvikio/_lib/libnvcomp.cxx:21155:185: warning: 'nvcomp::CascadedManager::CascadedManager(size_t, const nvcompBatchedCascadedOpts_t&, cudaStream_t, int, nvcomp::ChecksumPolicy, nvcomp::BitstreamKind)' is deprecated [-Wdeprecated-declarations]
  21155 |   __pyx_v_self->__pyx_base._impl = ((nvcomp::nvcompManagerBase *)new nvcomp::CascadedManager(__pyx_t_2, nvcompBatchedCascadedDefaultOpts, ((cudaStream_t)((void *)0)), __pyx_v_device_id));
        |                                                                                                                                                                                         ^
  In file included from /home/fstrug/.conda/envs/kvikio-dev-clone/include/nvcomp/nvcompManagerFactory.hpp:21,
                   from /home/fstrug/kvikio/python/kvikio/build/cp312-cp312-linux_x86_64/kvikio/_lib/libnvcomp.cxx:1257:
  /home/fstrug/.conda/envs/kvikio-dev-clone/include/nvcomp/cascaded.hpp:48:18: note: declared here
     48 |   [[deprecated]] CascadedManager(
        |                  ^~~~~~~~~~~~~~~
  /home/fstrug/kvikio/python/kvikio/build/cp312-cp312-linux_x86_64/kvikio/_lib/libnvcomp.cxx: In function 'int __pyx_pf_6kvikio_4_lib_9libnvcomp_16_GdeflateManager___cinit__(__pyx_obj_6kvikio_4_lib_9libnvcomp__GdeflateManager*, int, int, PyObject*, int)':
  /home/fstrug/kvikio/python/kvikio/build/cp312-cp312-linux_x86_64/kvikio/_lib/libnvcomp.cxx:21571:122: warning: 'nvcomp::GdeflateManager::GdeflateManager(size_t, const nvcompBatchedGdeflateOpts_t&, cudaStream_t, int, nvcomp::ChecksumPolicy, nvcomp::BitstreamKind)' is deprecated [-Wdeprecated-declarations]
  21571 |     __pyx_t_4 = new nvcomp::GdeflateManager(__pyx_v_chunk_size, __pyx_t_3, ((cudaStream_t)((void *)0)), __pyx_v_device_id);
        |                                                                                                                          ^
  In file included from /home/fstrug/.conda/envs/kvikio-dev-clone/include/nvcomp/nvcompManagerFactory.hpp:17,
                   from /home/fstrug/kvikio/python/kvikio/build/cp312-cp312-linux_x86_64/kvikio/_lib/libnvcomp.cxx:1257:
  /home/fstrug/.conda/envs/kvikio-dev-clone/include/nvcomp/gdeflate.hpp:53:18: note: declared here
     53 |   [[deprecated]] GdeflateManager(
        |                  ^~~~~~~~~~~~~~~
  /home/fstrug/kvikio/python/kvikio/build/cp312-cp312-linux_x86_64/kvikio/_lib/libnvcomp.cxx: In function 'int __pyx_pf_6kvikio_4_lib_9libnvcomp_11_LZ4Manager___cinit__(__pyx_obj_6kvikio_4_lib_9libnvcomp__LZ4Manager*, size_t, nvcompType_t, PyObject*, int)':
  /home/fstrug/kvikio/python/kvikio/build/cp312-cp312-linux_x86_64/kvikio/_lib/libnvcomp.cxx:21994:124: warning: 'nvcomp::LZ4Manager::LZ4Manager(size_t, const nvcompBatchedLZ4Opts_t&, cudaStream_t, int, nvcomp::ChecksumPolicy, nvcomp::BitstreamKind)' is deprecated [-Wdeprecated-declarations]
  21994 |     __pyx_t_4 = new nvcomp::LZ4Manager(__pyx_v_uncomp_chunk_size, __pyx_t_3, ((cudaStream_t)((void *)0)), __pyx_v_device_id);
        |                                                                                                                            ^
  In file included from /home/fstrug/.conda/envs/kvikio-dev-clone/include/nvcomp/nvcompManagerFactory.hpp:18,
                   from /home/fstrug/kvikio/python/kvikio/build/cp312-cp312-linux_x86_64/kvikio/_lib/libnvcomp.cxx:1257:
  /home/fstrug/.conda/envs/kvikio-dev-clone/include/nvcomp/lz4.hpp:45:18: note: declared here
     45 |   [[deprecated]] LZ4Manager(
        |                  ^~~~~~~~~~
  /home/fstrug/kvikio/python/kvikio/build/cp312-cp312-linux_x86_64/kvikio/_lib/libnvcomp.cxx: In function 'int __pyx_pf_6kvikio_4_lib_9libnvcomp_14_SnappyManager___cinit__(__pyx_obj_6kvikio_4_lib_9libnvcomp__SnappyManager*, size_t, PyObject*, int)':
  /home/fstrug/kvikio/python/kvikio/build/cp312-cp312-linux_x86_64/kvikio/_lib/libnvcomp.cxx:22371:148: warning: 'nvcomp::SnappyManager::SnappyManager(size_t, const nvcompBatchedSnappyOpts_t&, cudaStream_t, int, nvcomp::ChecksumPolicy, nvcomp::BitstreamKind)' is deprecated [-Wdeprecated-declarations]
  22371 |     __pyx_t_1 = new nvcomp::SnappyManager(__pyx_v_uncomp_chunk_size, nvcompBatchedSnappyDefaultOpts, ((cudaStream_t)((void *)0)), __pyx_v_device_id);
        |                                                                                                                                                    ^
  In file included from /home/fstrug/.conda/envs/kvikio-dev-clone/include/nvcomp/nvcompManagerFactory.hpp:19,
                   from /home/fstrug/kvikio/python/kvikio/build/cp312-cp312-linux_x86_64/kvikio/_lib/libnvcomp.cxx:1257:
  /home/fstrug/.conda/envs/kvikio-dev-clone/include/nvcomp/snappy.hpp:45:18: note: declared here
     45 |   [[deprecated]] SnappyManager(
        |                  ^~~~~~~~~~~~~
  [21/27] Linking CXX shared module kvikio/_lib/libnvcomp.cpython-312-x86_64-linux-gnu.so
  [22/27] Building CXX object kvikio/_lib/CMakeFiles/buffer.dir/buffer.cxx.o
  [23/27] Linking CXX shared module kvikio/_lib/buffer.cpython-312-x86_64-linux-gnu.so
  [24/27] Building CXX object kvikio/_lib/CMakeFiles/file_handle.dir/file_handle.cxx.o
  [25/27] Linking CXX shared module kvikio/_lib/file_handle.cpython-312-x86_64-linux-gnu.so
  [26/27] Building CXX object kvikio/_lib/CMakeFiles/remote_handle.dir/remote_handle.cxx.o
  [27/27] Linking CXX shared module kvikio/_lib/remote_handle.cpython-312-x86_64-linux-gnu.so
  *** Installing project into wheel...
  -- Install configuration: "Release"
  -- Installing: /tmp/tmp2jghdh20/wheel/platlib/kvikio/_lib/arr.cpython-312-x86_64-linux-gnu.so
  -- Set runtime path of "/tmp/tmp2jghdh20/wheel/platlib/kvikio/_lib/arr.cpython-312-x86_64-linux-gnu.so" to "$ORIGIN"
  -- Installing: /tmp/tmp2jghdh20/wheel/platlib/kvikio/_lib/buffer.cpython-312-x86_64-linux-gnu.so
  -- Set runtime path of "/tmp/tmp2jghdh20/wheel/platlib/kvikio/_lib/buffer.cpython-312-x86_64-linux-gnu.so" to "$ORIGIN"
  -- Installing: /tmp/tmp2jghdh20/wheel/platlib/kvikio/_lib/defaults.cpython-312-x86_64-linux-gnu.so
  -- Set runtime path of "/tmp/tmp2jghdh20/wheel/platlib/kvikio/_lib/defaults.cpython-312-x86_64-linux-gnu.so" to "$ORIGIN"
  -- Installing: /tmp/tmp2jghdh20/wheel/platlib/kvikio/_lib/driver_properties.cpython-312-x86_64-linux-gnu.so
  -- Set runtime path of "/tmp/tmp2jghdh20/wheel/platlib/kvikio/_lib/driver_properties.cpython-312-x86_64-linux-gnu.so" to "$ORIGIN"
  -- Installing: /tmp/tmp2jghdh20/wheel/platlib/kvikio/_lib/file_handle.cpython-312-x86_64-linux-gnu.so
  -- Set runtime path of "/tmp/tmp2jghdh20/wheel/platlib/kvikio/_lib/file_handle.cpython-312-x86_64-linux-gnu.so" to "$ORIGIN"
  -- Installing: /tmp/tmp2jghdh20/wheel/platlib/kvikio/_lib/future.cpython-312-x86_64-linux-gnu.so
  -- Set runtime path of "/tmp/tmp2jghdh20/wheel/platlib/kvikio/_lib/future.cpython-312-x86_64-linux-gnu.so" to "$ORIGIN"
  -- Installing: /tmp/tmp2jghdh20/wheel/platlib/kvikio/_lib/libnvcomp.cpython-312-x86_64-linux-gnu.so
  -- Set runtime path of "/tmp/tmp2jghdh20/wheel/platlib/kvikio/_lib/libnvcomp.cpython-312-x86_64-linux-gnu.so" to "$ORIGIN"
  -- Installing: /tmp/tmp2jghdh20/wheel/platlib/kvikio/_lib/libnvcomp_ll.cpython-312-x86_64-linux-gnu.so
  -- Set runtime path of "/tmp/tmp2jghdh20/wheel/platlib/kvikio/_lib/libnvcomp_ll.cpython-312-x86_64-linux-gnu.so" to "$ORIGIN"
  -- Installing: /tmp/tmp2jghdh20/wheel/platlib/kvikio/_lib/remote_handle.cpython-312-x86_64-linux-gnu.so
  -- Set runtime path of "/tmp/tmp2jghdh20/wheel/platlib/kvikio/_lib/remote_handle.cpython-312-x86_64-linux-gnu.so" to "$ORIGIN"
  *** Making wheel...
  *** Created kvikio-24.12.0-cp312-cp312-linux_x86_64.whl
  Building wheel for kvikio (pyproject.toml) ... done
  Created wheel for kvikio: filename=kvikio-24.12.0-cp312-cp312-linux_x86_64.whl size=960285 sha256=748038d5c08610fa03c1261b0f01dd9c35e2185a595390b5985c8685521d4a41
  Stored in directory: /tmp/pip-ephem-wheel-cache-bx3jdl9m/wheels/eb/e6/0d/bbb481fa73c4d05f339553d3fb4fe13a29e3ce22d008a711f4
Successfully built kvikio
Installing collected packages: kvikio
Successfully installed kvikio-24.12.0
(kvikio-dev-clone) [fstrug@jupyter-fstrug kvikio]$ pytest python/kvikio/tests/
============================================================================================================================================= test session starts =============================================================================================================================================
platform linux -- Python 3.12.7, pytest-8.3.3, pluggy-1.5.0
rootdir: /home/fstrug/kvikio/python/kvikio
configfile: pyproject.toml
plugins: cov-5.0.0
collected 1459 items                                                                                                                                                                                                                                                                                          

python/kvikio/tests/test_async_io.py .......                                                                                                                                                                                                                                                            [  0%]
python/kvikio/tests/test_basic_io.py .................................................................................................................................................................................................................................................................. [ 18%]
....................................................................................................................................................................................................................................................................................................... [ 38%]
............................................................................................................................s.......................................................................................................................................................................... [ 58%]
..................................................................ssssss.................................................................                                                                                                                                                               [ 67%]
python/kvikio/tests/test_benchmarks.py ............                                                                                                                                                                                                                                                     [ 68%]
python/kvikio/tests/test_defaults.py s....                                                                                                                                                                                                                                                              [ 69%]
python/kvikio/tests/test_examples.py ...                                                                                                                                                                                                                                                                [ 69%]
python/kvikio/tests/test_http_io.py .................................................                                                                                                                                                                                                                   [ 72%]
python/kvikio/tests/test_numpy.py ...........................                                                                                                                                                                                                                                           [ 74%]
python/kvikio/tests/test_nvcomp.py .......................xxxxxxXX.........x..xxx.........x........                                                                                                                                                                                                     [ 78%]
python/kvikio/tests/test_nvcomp_codec.py ................................................................................................................................................                                                                                                               [ 88%]
python/kvikio/tests/test_s3_io.py .....................................................................................                                                                                                                                                                                 [ 94%]
python/kvikio/tests/test_version.py .                                                                                                                                                                                                                                                                   [ 94%]
python/kvikio/tests/test_zarr.py .............................................................................                                                                                                                                                                                          [100%]

=========================================================================================================================== 1438 passed, 8 skipped, 11 xfailed, 2 xpassed in 41.28s ===========================================================================================================================

@jakirkham jakirkham mentioned this issue Oct 23, 2024
Open
@jakirkham
Copy link
Member

Noticed we are not explicitly listing these dependencies in the Python Conda package. I don't think that is the issue as the builds still show the dependencies being installed and detected. That said, it does seem worth fixing. Submitted PR: #513

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
5 participants
@madsbk @vyasr @jakirkham @fstrug @KyleFromNVIDIA