Frontera Installation

[lhy11009]

I have followed the ASPECT wiki and install on Frontera. It seems the method of installing Deal.II is not working or I am making some mistakes. Here is the description of the issue. In general, it's related to the bfd linker.

2025-06-13 – Compile deal.II 9.6.0 with Intel 19.1.1 and Intel MPI 19.0.9 on Frontera

Command(s) used

sbatch job_frontera-normal-intel-19.1.1-impi-19.0.9.sh

File(s) Edited

job_frontera-normal-intel-19.1.1-impi-19.0.9.sh:
    Summary:
    This Slurm batch script submits a build job on Frontera using Intel 19.1.1 and Intel MPI 19.0.9.
    It sets up the environment with required modules, creates a local.cfg with MKL and deal.II options,
    and runs the Candi installer to build deal.II 9.6 with Trilinos, SUNDIALS, HDF5, and P4EST.

    Full contents:
        #!/bin/bash -l
        #SBATCH -N 1
        #SBATCH -n 56
        #SBATCH --threads-per-core=1
        #SBATCH --tasks-per-node=56
        #SBATCH -o task-%j.stdout
        #SBATCH -e task-%j.stderr
        #SBATCH -t 12:00:00
        #SBATCH --partition=small
        #SBATCH --switches=1
        #SBATCH --mail-user=hylli@ucdavis.edu
        #SBATCH -A EAR23027
        #SBATCH --job-name=build_dealii

        module reset
        module load cmake intel/19.1.1 impi/19.0.9

        >&2 echo "list of modules:"
        >&2 module list

        echo "MKL=ON" > local.cfg
        echo "MKL_DIR=$MKLROOT/lib/intel64" >> local.cfg
        echo "NATIVE_OPTIMIZATIONS=ON" >> local.cfg
        echo "DEAL_II_VERSION=master" >> local.cfg
        echo "BUILD_EXAMPLES=OFF" >> local.cfg
        echo "DEAL_II_CONFOPTS=\"-D DEAL_II_WITH_64BIT_INDICES=ON -D DEAL_II_COMPONENT_EXAMPLES=OFF -DDEAL_II_WITH_COMPLEX_VALUES=OFF -DDEAL_II_WITH_LAPACK=ON -D LAPACK_LIBRARIES='/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin/libmkl_intel_lp64.so;/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin/libmkl_intel_thread.so;/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin/libmkl_core.so;/opt/intel/compilers_and_libraries_2020.1.217/linux/compiler/lib/intel64_lin/libiomp5.so;-lm;-ldl' -D DEAL_II_LINKER_FLAGS='-fuse-ld=bfd'\"" >> local.cfg

        yes '' | ./candi.sh -j 56 --packages="once:p4est once:hdf5 once:sundials once:trilinos dealii" -p /work2/06806/hyli/frontera/Softwares/dealii/dealii-9.6.0-Native-32bit-candi-intel-19.1.1-impi-19.0.9-normal

Error Message(s)

Occurred during compilation of deal.II-master:
    [ 50%] Building CXX object CMakeFiles/CheckCompilerSetupExec.dir/dummy.cpp.o  
    /opt/intel/.../mpi/intel64/bin/mpicxx -qopenmp-simd -march=native -O0 -g -gdwarf-2 -grecord-gcc-switches -MD -MT CMakeFiles/CheckCompilerSetupExec.dir/dummy.cpp.o -MF CMakeFiles/CheckCompilerSetupExec.dir/dummy.cpp.o.d -o CMakeFiles/CheckCompilerSetupExec.dir/dummy.cpp.o -c .../dummy.cpp

    [100%] Linking CXX executable CheckCompilerSetupExec  
    .../cmake -E cmake_link_script CMakeFiles/CheckCompilerSetupExec.dir/link.txt --verbose=1

    Linker error:
    ld.bfd: .../libstratimikosbelos.so.14.4: undefined reference to `Tpetra::MultiVector<float, int, long long, ...>::randomize()`
    ld.bfd: .../libstratimikosbelos.so.14.4: undefined reference to `Tpetra::DistObject<float, int, long long, ...>::packAndPrepare(...)`
    ld.bfd: ... and many other missing symbols involving Tpetra::MultiVector and Amesos2::Lapack with float template instantiations and Kokkos::Serial backend

[lhy11009]

Here is a full log output of configuring deal.ii
candi_configure.log

[tiannh7]

Hi @lhy11009 ,

The linker errors you’re seeing likely come from MKL-related floating-point instantiation conflicts in Trilinos. When using MKL, you should disable floating-point support in Trilinos by setting:

-D Trilinos_ENABLE_FLOAT=OFF
-D Tpetra_INST_FLOAT=OFF
-D Tpetra_INST_COMPLEX_FLOAT=OFF

Try adjusting your build config accordingly, and it should resolve the undefined references.

Let me know if you need help with the config!

Ninghui

[lhy11009]

@tiannh7 Thanks for the reply! I’ll update the status once the job finishes on Frontera.

I have a quick follow-up question. I noticed that the installation instructions on Frontera (https://github.com/geodynamics/aspect/wiki/Installation-on-Frontera) list the packages as:

./candi.sh -j 28 --packages="once:p4est once:hdf5 once:sundials once:trilinos dealii" -p $WORK2/libs

It looks like OpenBLAS isn’t included here—has it become unnecessary for Trilinos, or is there an existing OpenBLAS library already available on Frontera?

[lhy11009]

@tiannh7
I have added the 3 options to DEAL_II_CONFORTS
echo "DEAL_II_CONFOPTS=\"-D DEAL_II_WITH_64BIT_INDICES=ON -D DEAL_II_COMPONENT_EXAMPLES=OFF -DDEAL_II_WITH_COMPLEX_VALUES=OFF -DDEAL_II_WITH_LAPACK=ON -D LAPACK_LIBRARIES='/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin/libmkl_intel_lp64.so;/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin/libmkl_intel_thread.so;/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin/libmkl_core.so;/opt/intel/compilers_and_libraries_2020.1.217/linux/compiler/lib/intel64_lin/libiomp5.so;-lm;-ldl' -D Trilinos_ENABLE_FLOAT=OFF -D Tpetra_INST_FLOAT=OFF -D Tpetra_INST_COMPLEX_FLOAT=OFF -D DEAL_II_LINKER_FLAGS='-fuse-ld=bfd'\"" >> local.cfg

Maybe this is not the right place. It seems this doesn't address the issue.
candi_configure.log

[tiannh7]

Did you rebuild Trilinos with the new flags after modifying the configuration, or are you still using the previously built version? If Trilinos was not rebuilt with the updated flags, the linker errors may persist because the changes won’t be applied to the old libraries.

[lhy11009]

I built it in a new and clean directory. But it might be helpful to repeat. Let me try that. Also, one option in the slurm script includes the library for LAPACK libraries explicitly, and in the wiki it mentions that using the bfd linker doesn't guarantee that mkl recognizes this library. Is it possible that other libraries are needed?

[lhy11009]

After doing some research into this issue and general problems related to building with MKL, I think we're definitely heading in the right direction. From what I understand, MKL defaults to supporting double precision, so we need to explicitly disable the use of single precision (float) in both Trilinos and deal.II. I'm curious why this step wasn't included in the original solution on the wiki.

So it might be

1. The issue with the way the flags are written in local.cfg
2. We also need to provide something like"-D DEAL_II_TRILINOS_WITH_FLOAT=OFF" to tell deal.II to not depend on Trilinos' float numbers and the related libraries.
3. Maybe the previous build I tried is not clean (although I create a new directory for the building of deal.ii)

@tiannh7 For now, can you also leave maybe a short comment on whether you think this is the right approach?

[lhy11009]

So I tried to run the same slurm script with a new installation directory, but the same error persists. So it shouldn't be 3.

In addition, I want to check whether the flags I set in the local.cfg file is functioning, but this seems like a confusing practice. For example, I search the slurm log file with "Trilinos_ENABLE_FLOAT" and find no results. The other flags set within DEAL_II_CONFOPTS also seem missing. There are indeed other "Trilinos_ENABLE_" flags appearing in the log file, on the other hand. So I wonder if this suggests the local.cfg file is not being processed correctly.

I think there must be another way to check whether these flags are set up correctly. I believe they should be parsed into the makefile of the packages.

[tiannh7]

Hi @lhy11009,

Sorry for the late reply!

Regarding the build configuration, I suggest you try the following setup in your local.cfg or candi.cfg:

MKL=ON
BLAS_DIR=${MKLROOT}/lib/intel64
LAPACK_DIR=${MKLROOT}/lib/intel64

This explicitly points to the MKL libraries and should help ensure the linker and build system pick them up correctly.
Also, make sure that the local.cfg file is actually being processed during your build. Sometimes CMake or the build scripts might not read it if it's not in the expected location or if environment variables override it.

If you still have issues with the float/double precision flags, double-check that those options are passed correctly through your build system (e.g., via DEAL_II_CONFOPTS) and appear in the CMake configuration summary or logs.

Let me know how it goes!

[lhy11009]

In a recent compilation, I modified the echo command of the DEAL_II_CONFOPTS variable in the script to

echo 'DEAL_II_CONFOPTS="-D DEAL_II_WITH_64BIT_INDICES=ON -D DEAL_II_COMPONENT_EXAMPLES=OFF -D DEAL_II_WITH_COMPLEX_VALUES=OFF -D DEAL_II_WITH_LAPACK=ON -D LAPACK_LIBRARIES=/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin/libmkl_intel_lp64.so;/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin/libmkl_intel_thread.so;/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin/libmkl_core.so;/opt/intel/compilers_and_libraries_2020.1.217/linux/compiler/lib/intel64_lin/libiomp5.so;-lm;-ldl -D Trilinos_ENABLE_FLOAT=OFF -D Tpetra_INST_FLOAT=OFF -D Tpetra_INST_COMPLEX_FLOAT=OFF -D DEAL_II_TRILINOS_WITH_FLOAT=OFF -D DEAL_II_LINKER_FLAGS=-fuse-ld=bfd"' >> local.cfg

it seems the dealii compilation doesn't start correctly with this information. It mentioned "No source or binary directory provided". In the installation folder, I found the tmp/unpack contains the deal.II-master directory with cmake files and source files. However, in the build/deal.II-master directory, there are candi_build, candi_configure candi_configure.log. Have you ever seen this issue before?

-- Installing: /work2/06806/hyli/frontera/Softwares/dealii/dealii-9.6.0-Native-32bit-candi-intel-19.1.1-impi-19.0.9-normal-06162025/trilinos-release-14-4-0/lib/cmake/tribits/common_tpls/find_modules/FindHDF5.cmake
-- Installing: /work2/06806/hyli/frontera/Softwares/dealii/dealii-9.6.0-Native-32bit-candi-intel-19.1.1-impi-19.0.9-normal-06162025/trilinos-release-14-4-0/lib/cmake/tribits/common_tpls/utils
-- Installing: /work2/06806/hyli/frontera/Softwares/dealii/dealii-9.6.0-Native-32bit-candi-intel-19.1.1-impi-19.0.9-normal-06162025/trilinos-release-14-4-0/lib/cmake/tribits/common_tpls/utils/ParseLibraryList.cmake
-- Installing: /work2/06806/hyli/frontera/Softwares/dealii/dealii-9.6.0-Native-32bit-candi-intel-19.1.1-impi-19.0.9-normal-06162025/trilinos-release-14-4-0/lib/cmake/tribits/common_tpls/utils/AddPackageDependency.cmake
-- Installing: /work2/06806/hyli/frontera/Softwares/dealii/dealii-9.6.0-Native-32bit-candi-intel-19.1.1-impi-19.0.9-normal-06162025/trilinos-release-14-4-0/lib/cmake/tribits/common_tpls/utils/AddImportedLibrary.cmake
-- Installing: /work2/06806/hyli/frontera/Softwares/dealii/dealii-9.6.0-Native-32bit-candi-intel-19.1.1-impi-19.0.9-normal-06162025/trilinos-release-14-4-0/lib/cmake/tribits/common_tpls/FindTPLBLAS.cmake
-- Installing: /work2/06806/hyli/frontera/Softwares/dealii/dealii-9.6.0-Native-32bit-candi-intel-19.1.1-impi-19.0.9-normal-06162025/trilinos-release-14-4-0/lib/cmake/tribits/common_tpls/FindTPLHYPRE.cmake
-- Installing: /work2/06806/hyli/frontera/Softwares/dealii/dealii-9.6.0-Native-32bit-candi-intel-19.1.1-impi-19.0.9-normal-06162025/trilinos-release-14-4-0/lib/cmake/tribits/common_tpls/FindTPLBinUtils.cmake
-- Installing: /work2/06806/hyli/frontera/Softwares/dealii/dealii-9.6.0-Native-32bit-candi-intel-19.1.1-impi-19.0.9-normal-06162025/trilinos-release-14-4-0/lib/cmake/tribits/common_tpls/FindTPLCGNSDependencies.cmake
-- Installing: /work2/06806/hyli/frontera/Softwares/dealii/dealii-9.6.0-Native-32bit-candi-intel-19.1.1-impi-19.0.9-normal-06162025/trilinos-release-14-4-0/lib/cmake/tribits/common_tpls/FindTPLCGNS.cmake
-- Installing: /work2/06806/hyli/frontera/Softwares/dealii/dealii-9.6.0-Native-32bit-candi-intel-19.1.1-impi-19.0.9-normal-06162025/trilinos-release-14-4-0/lib/cmake/tribits/common_tpls/FindTPLProjectLastLib.cmake
-- Installing: /work2/06806/hyli/frontera/Softwares/dealii/dealii-9.6.0-Native-32bit-candi-intel-19.1.1-impi-19.0.9-normal-06162025/trilinos-release-14-4-0/lib/cmake/tribits/Copyright.txt
^[[1;34mdeal.II: configuration with P4EST^[[0m
^[[1;34mdeal.II: configuration with HDF5^[[0m
^[[1;34mdeal.II: configuration with TRILINOS^[[0m
^[[1;34mdeal.II: configuration with SUNDIALS^[[0m
^[[1;32mFetching dealii master^[[0m
Your branch is up to date with 'origin/master'.
^[[1;32mBuilding dealii master^[[0m
CMake Warning:
No source or binary directory provided. Both will be assumed to be the
same as the current working directory, but note that this warning will
become a fatal error in future CMake releases.

CMake Error: The source directory "/work2/06806/hyli/frontera/Softwares/dealii/dealii-9.6.0-Native-32bit-candi-intel-19.1.1-impi-19.0.9-normal-06162025/tmp/build/deal.II-master" does not appear to contain CMakeLists.txt.

[lhy11009]

@tiannh7 I'll have time to look at this again tomorrow. In the meantime, maybe a quick look from you could help me narrow down the issue. It might be that changing the DEAL_II_CONFOPTS is a bad idea and causes.

Also, if you are available in the week, I am very happy to arrange a short Zoom meeting for 30 mins. I am available most of the week except Tuesday afternoon and Wednesday morning. Let me know your thoughts. Have a great day.

[tiannh7]

Thanks for your message. I think you should use candi.cfg and modify options like MKL inside it, rather than changing DEAL_II_CONFOPTS. That approach tends to be more reliable.

If you still have any issues, I’m happy to arrange a Zoom meeting to discuss. Please feel free to send me the Zoom details at tiannh77@gmail.com

[lhy11009]

Hi @tiannh7, thank you for your helpful suggestion! I've made some progress.

I updated my candi.cfg with the following entries:

BLAS_DIR=${MKLROOT}/lib/intel64
LAPACK_DIR=${MKLROOT}/lib/intel64

I also realized that I had previously set USE_64_BIT_INDICES=OFF in candi.cfg, while the equivalent option in DEAL_II_CONFOPTS was -D DEAL_II_WITH_64BIT_INDICES=ON. This inconsistency may have caused earlier issues, so I’ve corrected it.

Additionally, I noticed that my branch of Candi had diverged from the master branch, so I synced it back up.

Now, deal.II begins compiling successfully, and the float-related issue for the linker appears to be resolved. However, I’ve encountered a new error:

[ 55%] Building CXX object source/numerics/CMakeFiles/object_numerics_release.dir/vector_tools_project_qp.cc.o
/work2/06806/hyli/frontera/Softwares/dealii/dealii-9.6.1-Native-32bit-candi-intel-19.1.1-impi-19.0.9-normal-06212025-1/tmp/unpack/deal.II-v9.6.1/include/deal.II/lac/trilinos_tpetra_sparse_matrix.templates.h(192): internal error: bad pointer                                                                                                                                                                                                SparsityPattern trilinos_sparsity;

This seems to be another compatibility issue, possibly related to Tpetra. I would appreciate any insights or suggestions you might have regarding this.

Thanks again for your continued support!

[tiannh7]

Hi @lhy11009,

Thanks for the update.

The new error you encountered does indeed look like a compatibility issue related to Tpetra. You might want to try using the branch I have been working on that addresses some MKL and Tpetra related problems.

Please give it a try and see if it helps resolve the issue.

[lhy11009]

Hi @tiannh7

I followed your suggestion and the job is running on Frontera. I want add 2 comments that we may discuss later.

1. I found in the trilinos.package that

   -D Trilinos_ENABLE_FLOAT=ON

I am not sure whether this conflict with our setting of DEAL_II_CONFOPTS in the candi.cfg

-D Trilinos_ENABLE_FLOAT=OFF -D Tpetra_INST_FLOAT=OFF -D Tpetra_INST_COMPLEX_FLOAT=OFF

My intuition is that the DEAL_II_CONFOPTS variable doesn't affect the configuration of Trilinos. I have a test where I manually set
-D Trilinos_ENABLE_FLOAT=OFF
in the trilinos.package, but this doesn't affect the result so far (the Tpetra compatibility error)

2. When I looked at the trilinos.package, I think MKL there is handled in a block of

   if [ ${MKL} = ON ]; then
   cecho ${INFO} "trilinos: configuration with MKL"

      if [ -z "${BLAS_DIR}" ] && [ -z "${LAPACK_DIR}" ]; then
          cecho ${INFO} "trilinos: configuration with BLAS_DIR=${MKL_DIR}"
          cecho ${INFO} "trilinos: configuration with LAPACK_DIR=${MKL_DIR}"
          CONFOPTS="${CONFOPTS} \
            -D BLAS_LIBRARY_DIRS:STRING=${MKL_DIR} \
            -D LAPACK_LIBRARY_DIRS:STRING=${MKL_DIR}"
      fi
   
      # Trilinos will complain that MKL does not support HAVE_TEUCHOS_BLASFLOAT. See
      # the discussion in https://github.com/dealii/candi/pull/92 for more details.
      cecho ${INFO} "trilinos: disabling some Tpetra instantiations because you are using MKL"
   
      CONFOPTS="${CONFOPTS} \
        -D BLAS_LIBRARY_NAMES:STRING='mkl_core;mkl_sequential' \
        -D Tpetra_INST_FLOAT:BOOL=OFF \
        -D Tpetra_INST_COMPLEX_FLOAT:BOOL=OFF \
        -D LAPACK_LIBRARY_NAMES:STRING=mkl_intel_lp64"
      else
          if [ ! -z "${BLAS_LIB}" ]; then
              # We need to specify the full name if using openblas.package:
              cecho ${INFO} "trilinos: configuration with BLAS_LIB=${BLAS_LIB}"
              CONFOPTS="${CONFOPTS} \
                -D TPL_BLAS_LIBRARIES:STRING=${BLAS_LIB} \
                -D TPL_LAPACK_LIBRARIES:STRING=${BLAS_LIB}"
          fi
      fi
   

This seems to be missing in the dealii.package. My thought is that this could be added there (maybe you already handled that in your branch). In this way, the BLAS library and LAPACK library are automatically set.