Code failures with SYCL on Aurora

[baperry2]

Various PeleLMeX cases are failing when compiled with SYCL on Aurora at ALCF. 3D Hot Bubble, 2D & 3D FlameSheet, Taylor-Green all fail (and likely most others). The only case I've tested that works is 2D Hot Bubble.

Reproducing the error: I’m running on an interactive compute node with the only change to the default environment being:
module load cmake
Clone PeleLMeX and its dependencies:
git clone --recursive https://github.com/AMReX-Combustion/PeleLMeX.git
Go to the HotBubble case:
cd PeleLMeX/Exec/RegTests/HotBubble
Build third party libraries (SUNDIALS):
make TPL COMP=intel-llvm USE_SYCL=TRUE
Build PeleLMeX:
make -j DIM=3 COMP=intel-llvm USE_SYCL=TRUE
Try to run:
mpiexec -n 2 ./PeleLMeX3d.sycl.MPI.ex input.3d-regt
I get failures that look like:
x4219c5s3b0n0.hsn.cm.aurora.alcf.anl.gov: rank 0 died from signal 11
The error can also be obtained running the code in evaluate mode, which just initializes data then runs one of the affected kernels (and saves the results to a plot file if it gets that far):
mpiexec -n 2 ./PeleLMeX3d.sycl.MPI.ex input.3d-regt peleLM.run_mode=evaluate peleLM.evaluate_vars="velForce density"

Debugging Progress
The first error has been isolated to this line in the makeVelForces kernel:

  amrex::Real rho_lcl = 0.0;
  if (is_incomp != 0) {
    rho_lcl = rho_incomp;
  } else {
    rho_lcl = rho(i, j, k); // this line fails
  }

In the cases being run is_incomp is always 0, but this is determined at runtime during initialization. The code works as expected if it is just hard-coded to 0/false. Removing the conditional and just setting rho_lcl = rho(i, j, k); also works. Leaving the conditional in place but making a trivial modification like rho_lcl = rho(i, j, k) + 1e-15; also works. Defining is_incomp to be something that is always false, like (i< -10000) but gets evaluated at runtime, does not work.

The same general code pattern appears several places and if this one is modified so it doesn't fail, the code fails at later instances like here:

            Real soverrho =
              (incompressible) != 0 ? a_dt / rho : a_dt / rho_arr(i, j, k);

[ThomasHowarth]

@baperry2 I've just run the FlameSheet test on this machine, which appears to have the same GPUs as Aurora (Intel Data Center GPU Max 1550s). I used the latest compilers available on this machine, oneAPI version: Intel(R) oneAPI DPC++/C++ Compiler 2025.2.0 (2025.2.0.20250605), and the code ran fine. When testing previously, I used a 2024 compiler - which one are you using? Perhaps I have the same one, and I can also test.

[ThomasHowarth]

I do find that I can have things crash if I change the way I'm pinning the MPI processes to the GPUs, i.e. one MPI process per GPU tile or per GPU or how I set up my I_MPI_ environment variables. However, it didn't seem to crash in the same place you're seeing.

[jrood-nrel]

It appears the segfault is happening during JIT. When we use a newer runtime on Aurora the HotBubble case runs successfully. I think this will be the case if we were to use the 2025.2 compiler as well.

[thickAsABrick]

A build of PeleLMeX on Aurora, with oneapi/release/2025.0.5, for the HotBubble input.3d-regt example, gave me this error message:

amrex::Abort::0::ParallelFor: The program was built for 1 devices
Build program log for 'Intel(R) Data Center GPU Max 1550':
IGC: Internal Compiler Error: Termination request sent to the program
!!!!! !!!
SIGABRT

With a build of the same case with oneapi/release/2025.2.0 on an internal platform, I was able to get the same example working.

[ThomasHowarth]

The closest compiler I had to this is oneAPI version: Intel(R) oneAPI DPC++/C++ Compiler 2025.0.4 (2025.0.4.20241205), and I just ran the 3D HotBubble on SNG2 using this compiler, and it ran fine. Whatever is causing this error looks to be more of a machine issue or MPI configuration error.

[baperry2]

Thanks @ThomasHowarth and @thickAsABrick for looking into this. I think it has now been isolated to a system-specific configuration problem on Aurora.

For testing purposes, I created a minimal reproducer for the failure we are seeing in AMReX: https://github.com/baperry2/amrex/tree/aurora-fail-reproducer/Tests/GPU/AuroraFailReproducer

I've also created a version of PeleLMeX that works around the issue described above and can be run on Aurora (hard-coding out incompressible capability, and must be run with peleLM.do_temporals = 0):

• https://github.com/baperry2/PeleLMeX/tree/works-on-aurora
• For reacting cases, a modified PelePhysics that removes the clean_init_massfrac option must also be used: https://github.com/AMReX-Combustion/PelePhysics/tree/works-on-aurora

[jrood-nrel]

At the moment I think we are just waiting for the SDK currently in testing on Sunspot to make its way to Aurora. Testing on Sunspot shows our PeleLMeX cases running fine with MPI.