[ROOT636] GCC 14.3 Compilation Error on EL8/EL9 due to __glibcxx_assert_fail in __host__ __device__ context

[akritkbehera]

Many builds are failing with GCC 14.3 on EL8 and EL9 due to __glibcxx_assert expanding into a call to the host-only function __glibcxx_assert_fail() inside host device contexts
Error
HeterogeneousCore/AlpakaTest

/data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc14/external/gcc/14.3.1-724da22786638848892aa9ded8fcd995/lib/gcc/x86_64-redhat-linux-gnu/14.3.1/../../../../include/c++/14.3.1/span:286:2: error: reference to __host__ function '__glibcxx_assert_fail' in __host__ __device__ function
  286 |         __glibcxx_assert(__idx < size());
      |         ^
/data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc14/external/gcc/14.3.1-724da22786638848892aa9ded8fcd995/lib/gcc/x86_64-redhat-linux-gnu/14.3.1/../../../../include/c++/14.3.1/x86_64-redhat-linux-gnu/bits/c++config.h:614:12: note: expanded from macro '__glibcxx_assert'
  614 |       std::__glibcxx_assert_fail();                                     \
      |            ^
src/HeterogeneousCore/AlpakaTest/plugins/alpaka/testESAlgoAsync.dev.cc:29:25: note: called by 'operator()'
   29 |             c[i] = a.z()[i] + b[i];
      |                         ^
/data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc14/external/alpaka/1.3.0-e2fe03c9e1160a0f010a42aad29ddb30/include/alpaka/kernel/TaskKernelGpuUniformCudaHipRt.hpp:76:13: note: called by 'gpuKernel<(lambda at src/HeterogeneousCore/AlpakaTest/plugins/alpaka/testESAlgoAsync.dev.cc:21:9), alpaka::ApiHipRt, alpaka::AccGpuUniformCudaHipRt<alpaka::ApiHipRt, std::integral_constant<unsigned long, 1>, unsigned int>, std::integral_constant<unsigned long, 1>, unsigned int, cms::alpakatest::AlpakaESTestSoALayoutA<>::ConstViewTemplateFreeParams<128, false, true, true>, const int *, cms::alpakatest::AlpakaESTestSoALayoutD<>::ViewTemplateFreeParams<128, false, true, true>, int>'
   76 |             kernelFnObj(const_cast<TAcc const&>(acc), args...);
      |             ^
/data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc14/external/gcc/14.3.1-724da22786638848892aa9ded8fcd995/lib/gcc/x86_64-redhat-linux-gnu/14.3.1/../../../../include/c++/14.3.1/x86_64-redhat-linux-gnu/bits/c++config.h:608:3: note: '__glibcxx_assert_fail' declared here
  608 |   __glibcxx_assert_fail()
      |   ^
1 error generated when compiling for gfx1030.

Full Logs: Here
RecoLocalCalo/EcalRecProducers

  /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc14/external/gcc/14.3.1-724da22786638848892aa9ded8fcd995/lib/gcc/x86_64-redhat-linux-gnu/14.3.1/../../../../include/c++/14.3.1/span:286:2: error: reference to __host__ function '__glibcxx_assert_fail' in __host__ __device__ function
   286 |         __glibcxx_assert(__idx < size());
      |         ^
/data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc14/external/gcc/14.3.1-724da22786638848892aa9ded8fcd995/lib/gcc/x86_64-redhat-linux-gnu/14.3.1/../../../../include/c++/14.3.1/x86_64-redhat-linux-gnu/bits/c++config.h:614:12: note: expanded from macro '__glibcxx_assert'
  614 |       std::__glibcxx_assert_fail();                                     \
      |            ^
src/RecoLocalCalo/EcalRecProducers/plugins/alpaka/AmplitudeComputationCommonKernels.h:244:21: note: called by 'operator()'
  244 |             energies[inputCh] = 0;
      |                     ^
/data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc14/external/alpaka/1.3.0-e2fe03c9e1160a0f010a42aad29ddb30/include/alpaka/kernel/TaskKernelGpuUniformCudaHipRt.hpp:76:13: note: called by 'gpuKernel<alpaka_rocm_async::ecal::multifit::Kernel_prep_1d_and_initialize, alpaka::ApiHipRt, alpaka::AccGpuUniformCudaHipRt<alpaka::ApiHipRt, std::integral_constant<unsigned long, 1>, unsigned int>, std::integral_constant<unsigned long, 1>, unsigned int, EcalDigiSoALayout<>::ConstViewTemplateFreeParams<128, false, true, true>, EcalDigiSoALayout<>::ConstViewTemplateFreeParams<128, false, true, true>, EcalUncalibratedRecHitSoALayout<>::ViewTemplateFreeParams<128, false, true, true>, EcalUncalibratedRecHitSoALayout<>::ViewTemplateFreeParams<128, false, true, true>, EcalMultifitConditionsSoALayout<>::ConstViewTemplateFreeParams<128, false, true, true>, Eigen::Matrix<float, 10, 1> *, Eigen::Matrix<char, 10, 1> *, bool *, bool *, bool *, char *, Eigen::Matrix<char, 10, 1> *, bool, bool>'
   76 |             kernelFnObj(const_cast<TAcc const&>(acc), args...);
      |             ^
/data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc14/external/gcc/14.3.1-724da22786638848892aa9ded8fcd995/lib/gcc/x86_64-redhat-linux-gnu/14.3.1/../../../../include/c++/14.3.1/x86_64-redhat-linux-gnu/bits/c++config.h:608:3: note: '__glibcxx_assert_fail' declared here
  608 |   __glibcxx_assert_fail()

Full Logs: [Here](https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/el8_amd64_gcc14/CMSSW_16_0_ROOT636_X_2025-10-06-1400/
RecoLocalCalo/EcalRecProducers)

Reference: After introducing of call to std::__glibcxx_assert_fail() in _GLIBCXX_HAVE_IS_CONSTANT_EVALUATED ifdef block, clang fails to compile some code (found on pytorch)

All Errors: here

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @akritkbehera.

@Dr15Jones, @ftenchini, @makortel, @mandrenguyen, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

assign core, heterogeneous

[cmsbuild]

New categories assigned: core,heterogeneous

@Dr15Jones,@fwyzard,@makortel,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

[makortel]

IIUC the failing part is specifically ROCm compilation.

[dan131riley]

Apparently in the new version the expansion of __glibcxx_assert is no longer constexpr. There's some discussion in the libstdc++ bugzilla.

[fwyzard]

Looks like a workaround was merged in LLVM a few months ago (llvm/llvm-project#136133), but it's only in the LLVM 21.x branch.

Even ROCm 7.0 is still using LLVM 20, so it might not have the fix.

[makortel]

I came across with

HIP provides its own implementation for assert for usage in device code in hip/hip_runtime.h.

from https://rocm.docs.amd.com/projects/HIP/en/docs-develop/how-to/kernel_language_cpp_support.html

I wonder if we should avoid #include <cassert> for device code then? I see some in

cmssw/HeterogeneousCore/AlpakaInterface/interface/CachingAllocator.h

Line 4 in c6d61a7

#include <cassert>

cmssw/HeterogeneousCore/AlpakaInterface/interface/getDeviceCachingAllocator.h

Line 4 in c6d61a7

#include <cassert>

that would get included in the compilation of testESAlgoAsync.dev.cc.

[smuzaffar]

hopefully cms-sw/cmsdist#10138 should fix this issue.