[bug] Warning: CCTag library is built without CUDA support, so we can't enable CUDA.

[hofbauerc]

Hello,

I am trying to build the library using CUDA and everything seems to be ok during the build process. However, when I try to do a simple detection using CUDA I get the following output:
Warning: CCTag library is built without CUDA support, so we can't enable CUDA.

It seems that the CCTAG_WITH_CUDA flag is not set as seen in the message print.

I work inside a custom docker container with the following base image:
FROM nvidia/cuda:12.3.1-devel-ubuntu22.04

I use the newest versions of TBB, eigen3, cmake and boost as well as opencv 4.7

I use following flags for building:
cmake -DCCTAG_BUILD_TESTS:BOOL=ON -DCCTAG_BUILD_APPS:BOOL=ON -DBUILD_SHARED_LIBS:BOOL=ON -DCCTAG_WITH_CUDA:BOOL=ON ..

And the output looks good as well:

-- The C compiler identification is GNU 11.4.0
-- The CXX compiler identification is GNU 11.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Build type not set, building in Release configuration
-- Found Boost: /usr/lib/x86_64-linux-gnu/cmake/Boost-1.74.0/BoostConfig.cmake (found suitable version "1.74.0", minimum required is "1.66.0") found components: atomic chrono date_time filesystem program_options serialization system thread timer math_c99 stacktrace_basic unit_test_framework 
-- Found Boost: version 1.74.0
-- Try finding CUDA
-- BUILD_SHARED_LIBS ON
-- Found CUDA: /usr/local/cuda (found suitable version "12.3", minimum required is "9.0") 
-- Building in release mode
-- Found OpenCV: /opt (found version "4.7.0") found components: core videoio imgproc imgcodecs 
-- Found Eigen: version 3.4.0
-- Found TBB: version 2021.5.0
-- CCTAG already there
-- BOOST already there
-- Found OpenCV: /opt (found version "4.7.0") found components: core videoio imgproc imgcodecs highgui 
-- TBB already there
-- Found DevIL: /usr/lib/x86_64-linux-gnu/libIL.so  
-- DevIL found


******************************************
Building configuration:

-- CCTag version: 1.0.4
-- Build type: Release
-- Build Shared libs: ON
-- Build applications: ON
-- Build tests: ON
-- Build documentation: OFF
-- Cuda support: ON
-- Compiling for CUDA CCs: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_52,code=sm_52;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_89,code=sm_89;-gencode;arch=compute_90,code=sm_90;-gencode;arch=compute_90,code=compute_90
-- Enable Eigen alignment: OFF
-- Enable AVX2 optimizations: OFF
-- [debug] Serialize all the output: OFF
-- [debug] Enable visual debug: OFF
-- [debug] Disable output stream: ON
-- [debug] nvcc additional warnings: OFF
-- Install path: /usr/local

******************************************


-- Configuring done
-- Generating done
-- Build files have been written to: /opt/CCTag/build

And this would be the CMakeLists.txt for my detection cpp file:

# CMakeLists.txt
cmake_minimum_required(VERSION 3.13)

# Project name and version
project(cctag_detection VERSION 1.0)

# Specify the C++ standard
set(CMAKE_CXX_STANDARD 14)
set(CMAKE_CXX_STANDARD_REQUIRED True)

# Find OpenCV and Boost
find_package(OpenCV 4.7.0 REQUIRED)
find_package(Boost REQUIRED COMPONENTS filesystem timer)
find_package(CUDA REQUIRED)
find_package(CCTag REQUIRED)

# Include directories for CCTag
include_directories(${OpenCV_INCLUDE_DIRS} ${Boost_INCLUDE_DIRS})
include_directories(/usr/local/lib/)
include_directories(/usr/include/eigen3)
include_directories(/usr/local/cuda/include)
include_directories(${CUDA_INCLUDE_DIRS})

# Link directories for CCTag
link_directories(/usr/local/lib/)

# Add the executable
add_executable(cctag_detection cctag_detector.cpp)

# Link libraries
target_link_libraries(cctag_detection PRIVATE CCTag::CCTag ${OpenCV_LIBS} ${Boost_LIBRARIES} ${CUDA_LIBRARIES})

Which also builds without issues.

I am using the newest CCTag release from a few weeks ago.

Am I missing some flag that needs to be set? Are there some CUDA version issues with 12.3.1?

[simogasp]

Are you running docker with nvidia runtime, i.e. with docker run --rm --runtime=nvidia ... as explained here?
You need to have the nvidia containe toolkit https://github.com/NVIDIA/nvidia-container-toolkit?tab=readme-ov-file to be able to access the GPU with docker.

[hofbauerc]

Hi,

Yeah sure, I use my GPU (in this case a RTX 4060 Laptop Version) all the time in multiple docker environments. The GPU is listed inside the docker and all the stuff that requires one runs on it, I already checked that. It somehow seems that the flag for if the code was compiled using CUDA is not set. Could be something else but it seems like it.

Do you have any other ideas what the issue could be?

Thanks in advance.

[simogasp]

Maybe for some reason it's not including the config file.
You can try to add

#include <cctag/cctag_config.hpp>

at the top of src/applications/detection/CmdLine.hpp and main.cpp in the same folder and recompile.

This file is the one who defines (or not) CCTAG_WITH_CUDA used in the code.

[hofbauerc]

Thanks for the info, sadly it did not fix my issue.

Any other ideas what could cause that?

My docker environment in terms of GPU seems fine, the building process also runs without issues.

[simogasp]

ah sorry I overlooked you had your own detection file, I was thinking you were using the one in the library.
Can you try to use that one? It is in the build directory under something like ./Linux/detection.
Does that work with cuda?
And in any case, you should include the cctag_config.hpp in your file as well.

Just a side note, unrelated to the issue, you can remove these lines from your cmakelists

# Include directories for CCTag
include_directories(${OpenCV_INCLUDE_DIRS} ${Boost_INCLUDE_DIRS})
include_directories(/usr/local/lib/)
include_directories(/usr/include/eigen3)
include_directories(/usr/local/cuda/include)
include_directories(${CUDA_INCLUDE_DIRS})

# Link directories for CCTag
link_directories(/usr/local/lib/)

everything is taken in charge by target_link_libraries, adding those manually may result in conflicts.

[hofbauerc]

Hi,

I just tested the detection application from the code itself, however there seems to bee some boost issue. Could this be related to the CUDA issue?

[simogasp]

I just tested the detection application from the code itself, however there seems to bee some boost issue. Could this be related to the CUDA issue?

No it is not.
Add --use-cuda to the command line to ask for cuda, at least we can see if we hit the same warning.
Do you have writing permissions in the folder I suppose? the exception seems like to indicate it cannot create a file.

[hofbauerc]

Yeah sorry there were some permission issues, could have thought about that.

I tried to force it with --use-cuda but still the same issue.

[simogasp]

can you ldd detection and then ldd libCCTag.so to see if it is linked to cuda?

[hofbauerc]

Yea seems to be linked correctly.

[simogasp]

Try to modify src/CMakeLists.txt and add right before line 182

  target_compile_definitions(CCTag PUBLIC CCTAG_WITH_CUDA)

Add the same for the detection target in its cmakelists

  target_compile_definitions(detection PUBLIC CCTAG_WITH_CUDA)

and rebuild everything

[hofbauerc]

Hi,

I tried the above and now the CUDA error is gone, but the detection using GPU seems to not work properly. Here both with and without CUDA:

[theDigitalGuy]

I had a very similar issue. I had to do the following in order to actually compile it with cuda support.

Try to modify src/CMakeLists.txt and add right before line 182

target_compile_definitions(CCTag PUBLIC CCTAG_WITH_CUDA)

Add the same for the detection target in its cmakelists

target_compile_definitions(detection PUBLIC CCTAG_WITH_CUDA)

and rebuild everything

For my cmake version 3.28.3 I also hat to add the follow to my cmake file:

if (POLICY CMP0146)
  cmake_policy(SET CMP0146 OLD)
endif()

The compute capability level of my GPU is 8.6 and I then got the following error as I tested the example detection (./Linux-x86_64/detection -n 3 -i ../sample/01.png -s --use-cuda) with the -s flag :

cctag/cuda/frame_07c_eval.cu:228
    cudaGetLastError failed: too many resources requested for launch
terminate called recursively
terminate called recursively
terminate called after throwing an instance of 'thrust::system::system_error'
  what():  get_max_shared_memory_per_block :failed to cudaGetDevice: cudaErrorCudartUnloading: driver shutting down
Aborted (core dumped)

To me it seems like the line dim3 block( 32, 32, 1 ); in frame_07c_eval.cu is causing the issue. By looking at the following chart (https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#features-and-technical-specifications-feature-support-per-compute-capability) it seems possible that the CCTag library was developed on cards that supported 32 resident blocks per SM and some generations later sometimes only 16.

This workaround (develop...theDigitalGuy:CCTag:debugCudaProblem) seems to work okay. Seldom I get something like the following:
cctag/cuda/tag_identify.cu:545 imageCenterOptLoop is called from CPU code with vCutSize 21 instead of 22

My work around is probably not very elegant, but it hopefully helps to solve the problem properly.
Maybe something like

    // Determine block configuration
    int max_threads_per_block = min(prop.maxThreadsPerBlock, 512);
    int threads_per_dim = static_cast<int>(sqrt(max_threads_per_block));

    // Ensure threads_per_dim is a power of 2 and <= 32
    threads_per_dim = min(32, static_cast<int>(pow(2, floor(log2(threads_per_dim)))));

could be used to determine the optimal thread count.