Compiling on a Cray / MI300A system

[joneuhauser]

I'm trying to compile on Hunter, the new APU cluster in Stuttgart (HLRS). I'm trying to compile the next version (5152fec).

Currently Loaded Modules:
  1) craype-x86-genoa     4) craype/2.7.32       7) cray-pals/1.3.2     10) xpmem/2.3                13) rocm/6.2.2                 16) cray-mpich/8.1.30
  2) libfabric/1.20.1     5) cray-dsmml/0.3.0    8) system/site_names   11) perftools-base/24.07.0   14) HLRS/APU/testing-20250305  17) cray-libsci/24.07.0
  3) craype-network-ofi   6) PrgEnv-cray/8.5.0   9) system/ws/20250205  12) craype-accel-amd-gfx942  15) cce/18.0.1                 18) system/wrappers/1.0

Here's my attempts so far:

• module load PrgEnv-Cray && export CC=cc FC=ftn CXX=CC:

  CMake Error at cmake/utils.cmake:50 (message):
    Cannot identify underlying compiler used by
    /opt/cray/pe/craype/2.7.32/bin/CC```
  

• module load PrgEnv-Cray && export CC= FC= CXX=:

  CMake Error at cmake/nek5000.cmake:15 (message):
    Fortran compiler Cray not supported to build Nek5000 interface!
  

  If I disable that check (cray should be supported by nek5000, at least that's my understanding of makenek.inc can't detect compiler version on cray Nek5000#17), Configure is succesful, but it fails on building:

     [ 21%] Building Fortran object 3rd_party/lapack/BLAS/SRC/CMakeFiles/blas_64_obj.dir/blas_64_obj/dcopy.f.o
  ftn-78 ftn: ERROR in command line
    The -m option has an invalid argument, "arch=native".
  ftn-78 ftn: ERROR in command line
    The -m option has an invalid argument, "tune=native".
  ftn-78 ftn: ERROR in command line
    The -f option has an invalid argument, "tree-vectorize".
  

• module load PrgEnv-gnu && export CC=cc FC=ftn CXX=CC: configures sucessfully
  Warning:

  configure: ***********************************************************************  
  configure: Configuring with --with-hip=yes without unified memory.
  configure: It only works for structured solvers and selected unstructured solvers
  configure: Use --enable-unified-memory to compile with unified memory.
  configure: ***********************************************************************
  

  and various errors:

  g++: error: language hip not recognized
  

• module load PrgEnv-gnu && export CC= FC= CXX=: same as above
• module load PrgEnv-gnu-amd && export CC=cc FC=ftn CXX=CC:

  CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
    Could NOT find OpenMP_C (missing: OpenMP_C_FLAGS OpenMP_C_LIB_NAMES)
  

• module load PrgEnv-amd && export CC=cc FC=ftn CXX=CC:

  CMake Error at /usr/share/cmake/Modules/CMakeTestFortranCompiler.cmake:59 (message):
    The Fortran compiler
  
      "/opt/cray/pe/craype/2.7.32/bin/ftn"
  
    is not able to compile a simple test program.
  
    It fails with the following output:
  
      Change Dir: [HOME]/NekRS/build/CMakeFiles/CMakeScratch/TryCompile-gw04rp
    
      Run Build Command(s):/usr/bin/cmake -E env VERBOSE=1 /usr/bin/gmake -f Makefile cmTC_c234e/fast && /usr/bin/gmake  -f CMakeFiles/cmTC_c234e.dir/build.make CMakeFiles/cmTC_c234e.dir/build
      gmake[1]: Entering directory '[HOME]/NekRS/build/CMakeFiles/CMakeScratch/TryCompile-gw04rp'
      Building Fortran object CMakeFiles/cmTC_c234e.dir/testFortranCompiler.f.o
      /opt/cray/pe/craype/2.7.32/bin/ftn    -c [HOME]/NekRS/build/CMakeFiles/CMakeScratch/TryCompile-gw04rp/testFortranCompiler.f -o CMakeFiles/cmTC_c234e.dir/testFortranCompiler.f.o
      Linking Fortran executable cmTC_c234e
      /usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_c234e.dir/link.txt --verbose=1
      /opt/cray/pe/craype/2.7.32/bin/ftn CMakeFiles/cmTC_c234e.dir/testFortranCompiler.f.o -o cmTC_c234e 
      ld.lld: error: non-exported symbol '__divdc3' in '/opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/libgcc.a(_divdc3.o)' is referenced by DSO '/opt/cray/pe/libsci/24.07.0/AMD/6.0/x86_64/lib/libsci_amd_mpi.so'
  

• module load PrgEnv-amd && export CC= FC= CXX= (which uses AMD Flang): Build and install runs fine. On running the turbPipePeriodic test case (with MPICH_GPU_SUPPORT_ENABLED=1 exported in the submit script), I get the following error:

  aborting job:
  MPIDI_CRAY_init: GPU_SUPPORT_ENABLED is requested, but GTL library is not linked
  

  I think this should be fixed by compiling with the Cray compiler wrappers, but that didn't work, see above.

I currently don't have the new AMD Fortran compiler (amdflang-new) as a module, so I can't test easily.

Has someone else succeded to compile on such a system (LUMI is probably similar)?

Thanks in advance!

[joneuhauser]

Using module load PrgEnv-amd && export CC= FC= CXX=
and then build with : ./build.sh -DCMAKE_EXE_LINKER_FLAGS=\"-L${MPICH_DIR}/lib -lmpi ${PE_MPICH_GTL_DIR_amd_gfx942} ${PE_MPICH_GTL_LIBS_amd_gfx942}\"
gets me until compiling nek5000 at least, but fails with

ld.lld: error: /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/crtbeginS.o:(.text+0x76): relocation R_X86_64_PC32 out of range: 14016610053 is not in [-2147483648, 2147483647]; references section '.bss'
>>> referenced by crtstuff.c

[joneuhauser]

Compiling nek5000 with -mcmodel=large doesn't help

[joneuhauser]

With the Cray compiler environment, I can finally build. Oddities that I needed to fix:

• The Cray C compiler identifies as clang, but clang is not actually a standalone executable. So the check in utils.cmake failed.
• HYPRE wants to be configured with --enable-unified-memory on HIP.
• The Cray C compiler supports -mcmodel=medium, but not crayftn. So the fortran compiler needs different flags in this case.
• target_link_libraries(libocca PRIVATE OpenMP::OpenMP_CXX) does not return any linker flags, but -fopenmp is required to link. Hence, I build with ./build.sh -DCMAKE_EXE_LINKER_FLAGS="-fopenmp".

[stgeke]

Officially, we do not support the Cray compilers. I recommend using the AMD compilers instead, as they are better supported in our environment. The linking error you encountered suggests that libgcc was not built with a mcmodel compatible with the current compilation settings we're using for nek5000.

[stgeke]

Are you using the next branch? Make sure to unload cray-libsci to avoid a symbol name conflict.

[joneuhauser]

Tried unloading cray-libsci, but now I get (after rebuilding everything)

/NekRS/examples/turbPipePeriodic/.cache/nek5000/libturbPipe.so: undefined symbol: _dger_

[joneuhauser]

Seems not, at least I can't reproduce that problem anymore. Progress! Now I get (turbPipePeriodic test case):

regenerate geometry data 1


    0  ERROR:  Vanishing Jacobian near     68th node of element        38.
 2.91256942102234664E-6,  -1.08718889675616786E-4
    0 xyz:   1.60767E-02   3.78673E-01   1.09937E-02
    0 xyz:   3.09877E-02   3.77091E-01   1.09937E-02

Let me try a different test case.

[joneuhauser]

Same error with the channel test case.

[joneuhauser]

Opening the resulting xyzturbPipe.fld01 file confirms that something goes very wrong with the mesh:

[stgeke]

For now, I suggest removing the mcmodel flag from cmake/nek5000.cmake before building the code. The downside is that this will significantly limit the number of elements per MPI rank, which may be too small to achieve good GPU performance. However, it could still be worthwhile to run some basic tests to ensure everything is functioning correctly.

[joneuhauser]

Alright, without mcmodel and using the AMD compilers, without wrappers, but extra flags to find the accelerators, the nek5000 compilation works for the channel case, but turbPipePeriodic already fails (7.5k elements).

In any case, now it hangs trying to build the udf (doesn't even crash). That step worked with the Cray compiler.

[stgeke]

You can try to run the case with verbose = true in par (GENERAL section). This might help to understand why it seems to hang. Did you build the code on a compute node? I suggest to use compiler wrappers (e.g. cc, CC and ftn).

[joneuhauser]

I suggest to use compiler wrappers (e.g. cc, CC and ftn).

This did not work (as mentioned initially, and unloading libsci doesn't help here)

CMake Error at /usr/share/cmake/Modules/CMakeTestFortranCompiler.cmake:59 (message):
  The Fortran compiler

    "/opt/cray/pe/craype/2.7.32/bin/ftn"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: [HOME]/NekRS/build/CMakeFiles/CMakeScratch/TryCompile-gw04rp
  
    Run Build Command(s):/usr/bin/cmake -E env VERBOSE=1 /usr/bin/gmake -f Makefile cmTC_c234e/fast && /usr/bin/gmake  -f CMakeFiles/cmTC_c234e.dir/build.make CMakeFiles/cmTC_c234e.dir/build
    gmake[1]: Entering directory '[HOME]/NekRS/build/CMakeFiles/CMakeScratch/TryCompile-gw04rp'
    Building Fortran object CMakeFiles/cmTC_c234e.dir/testFortranCompiler.f.o
    /opt/cray/pe/craype/2.7.32/bin/ftn    -c [HOME]/NekRS/build/CMakeFiles/CMakeScratch/TryCompile-gw04rp/testFortranCompiler.f -o CMakeFiles/cmTC_c234e.dir/testFortranCompiler.f.o
    Linking Fortran executable cmTC_c234e
    /usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_c234e.dir/link.txt --verbose=1
    /opt/cray/pe/craype/2.7.32/bin/ftn CMakeFiles/cmTC_c234e.dir/testFortranCompiler.f.o -o cmTC_c234e 
    ld.lld: error: non-exported symbol '__divdc3' in '/opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/libgcc.a(_divdc3.o)' is referenced by DSO '/opt/cray/pe/libsci/24.07.0/AMD/6.0/x86_64/lib/libsci_amd_mpi.so'

[joneuhauser]

But that looks like the PrgEnv is incorrectly setup, not a problem with something that I'm doing.

[joneuhauser]

Fixed by export FFLAGS="-lgcc". Let's see how far I get with that one!

[stgeke]

Did you delete .cache?

…

On Fri, 11 Apr 2025 at 12:27, Jonathan Neuhauser ***@***.***> wrote: Tried unloading cray-libsci, but now I get (after rebuilding everything) /NekRS/examples/turbPipePeriodic/.cache/nek5000/libturbPipe.so: undefined symbol: _dger_ — Reply to this email directly, view it on GitHub <#634 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEN2M75NMOHOKRQHFBECWJT2Y6KI5AVCNFSM6AAAAAB23LNIXKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEOBQGIZTOMY> . You are receiving this because you commented.Message ID: ***@***.***>

[joneuhauser]

Partial success! I finally get it to compile and run with

export FFLAGS="-lgcc"
export CC=cc FC=ftn CXX=CC
module load PrgEnv-amd
module unload cray-libsci
cmake -S . -B build -Wfatal-errors
cmake --build ./build --target install -j24

The channel case now completes sucessfully. The turbPipePeriodic fails because of the mcmodel issue.

[joneuhauser]

I switched to a different linker to circumvent the memory model issue.

export FFLAGS="-fuse-ld=bfd"

Now I can run the turbPipePeriodic case with the original settings. --cimode=1 timings:

  solve                 3.16591e+01s  100.0      
    min                 9.43427e-02s
    max                 8.12869e+00s
    flops/rank          1.15927e+12
    checkpointing       5.65680e-01s   1.8        1
    udfExecuteStep      1.60344e-03s   0.0        200
    makef               3.41827e+00s  10.8        200
      udfUEqnSource     5.98709e-03s   0.0   0.2  200
    udfProperties       8.75081e-04s   0.0        201
    velocitySolve       6.08549e+00s  19.2        200
      rhs               1.39646e-01s   0.4   2.3  200
    pressureSolve       2.11087e+01s  66.7        200
      rhs               3.23801e-01s   1.0   1.5  200
      preconditioner    1.58768e+01s  50.1  75.2  2271
        pMG smoother    1.15140e+01s  36.4  72.5  4542
        pMG smoother    1.99949e+00s   6.3  12.6  4542
        coarse grid     1.32561e+00s   4.2   8.3  2271
      initial guess     3.13625e-01s   1.0   1.5  200

What case do you have comparison for regarding inter-/intra-node MPI performance?

[stgeke]

It might work now with mcmodel = medium as libgcc is different.

…

On Fri, 11 Apr 2025 at 14:02, Jonathan Neuhauser ***@***.***> wrote: Partial success! I finally get it to compile and run with export FFLAGS="-lgcc" export CC=cc FC=ftn CXX=CC module load PrgEnv-amd module unload cray-libsci cmake -S . -B build -Wfatal-errors cmake --build ./build --target install -j24 The channel case now completes sucessfully. The turbPipePeriodic fails because of the mcmodel issue. — Reply to this email directly, view it on GitHub <#634 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEN2M73ITZJ4JDFZHFKKJML2Y6VODAVCNFSM6AAAAAB23LNIXKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEOBQGMZDGNQ> . You are receiving this because you commented.Message ID: ***@***.***>

[joneuhauser]

Thanks. I tried that (and verbose=True confirms that ftn is called with that argument) but linking still fails with the same error.

[joneuhauser]

To summarize the solution:

• Use the AMD compiler with compiler wrappers.
• Compile using

  rm -r build || true
  rm -r .local/nekrs || true
  export FFLAGS=""
  export CC=cc FC=ftn CXX=CC
  module load PrgEnv-amd
  module unload cray-libsci
  cmake -S . -B build -Wfatal-errors
  cmake --build ./build --target install -j24
  echo "Compiling complete"
  

• Run using

  module load PrgEnv-amd
  module unload cray-libsci
  export FFLAGS="-fuse-ld=bfd"
  export NEKRS_HOME=$HOME/.local/nekrs
  export PATH=$HOME/Nek5000/bin:$NEKRS_HOME/bin:$PATH
  export CC=cc FC=ftn CXX=CC
  export MPICH_GPU_SUPPORT_ENABLE=1
  mpirun ...
  

[joneuhauser]

For the master branch, the following patch allows me to sucessfully build and run with the above modules and environment variables (mostly backports from the next branch, plus some small cmake changes):

compile-with-amdflang-patch.txt

These changes are not necessary for the next branch.

EDIT: Crash reading the .re2 fixed by another backport.

[joneuhauser]

Performance results of turbPipePeriodic on master, single MI300A:

  solve                 9.91816e+00s  100.0      
    min                 3.51631e-02s
    max                 6.19750e-01s
    flops/rank          5.78939e+11
    checkpointing       4.95392e-01s   5.0        1
    udfExecuteStep      1.34529e-03s   0.0        200
    makef               1.04832e+00s  10.6        200
    udfProperties       1.42409e-03s   0.0        201
    velocitySolve       7.27806e-01s   7.3        202
      rhs               3.96003e-02s   0.4   5.4  202
      preconditioner    2.75881e-02s   0.3   3.8  949
    pressureSolve       7.23562e+00s  73.0        202
      rhs               1.13484e-01s   1.1   1.6  202
      preconditioner    6.38645e+00s  64.4  88.3  865
        pMG smoother    2.66106e+00s  26.8  41.7  1730
        pMG smoother    2.36962e+00s  23.9  37.1  1730
        coarse grid     7.77636e-01s   7.8  12.2  865
      initial guess     1.56117e-01s   1.6   2.2  202

That is less than a third of the Next branch in total time, min timestep, makef, pressure solve and almost a factor of 10 for velocity solve?

[stgeke]

Please DM me.

…

On Mon, 14 Apr 2025 at 11:27, Jonathan Neuhauser ***@***.***> wrote: I switched to a different linker to circumvent the memory model issue. export FFLAGS="-fuse-ld=bfd" Now I can run the turbPipePeriodic case with the original settings. --cimode=1 timings: solve 3.16591e+01s 100.0 min 9.43427e-02s max 8.12869e+00s flops/rank 1.15927e+12 checkpointing 5.65680e-01s 1.8 1 udfExecuteStep 1.60344e-03s 0.0 200 makef 3.41827e+00s 10.8 200 udfUEqnSource 5.98709e-03s 0.0 0.2 200 udfProperties 8.75081e-04s 0.0 201 velocitySolve 6.08549e+00s 19.2 200 rhs 1.39646e-01s 0.4 2.3 200 pressureSolve 2.11087e+01s 66.7 200 rhs 3.23801e-01s 1.0 1.5 200 preconditioner 1.58768e+01s 50.1 75.2 2271 pMG smoother 1.15140e+01s 36.4 72.5 4542 pMG smoother 1.99949e+00s 6.3 12.6 4542 coarse grid 1.32561e+00s 4.2 8.3 2271 initial guess 3.13625e-01s 1.0 1.5 200 What case do you have comparison for regarding inter-/intra-node MPI performance? — Reply to this email directly, view it on GitHub <#634 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEN2M77PB4S56INQ55YJJJT2ZN5R5AVCNFSM6AAAAAB23LNIXKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEOBSGY4TCMY> . You are receiving this because you commented.Message ID: ***@***.***>