Performance issue on Adastra

[tpadioleau]

When running the unit tests on the MI250X of Adastra I could notice a significant slow down compared to V100/A100 on an other computer. Here are some numbers:

364/555 Test #364: Plans1D.1DFFT_1DView<std::pair<float, Kokkos::LayoutLeft>> ..........................................................................   Passed   31.96 sec
        Start 367: Plans1D.1DFFT_1DView<std::pair<float, Kokkos::LayoutRight>>
365/555 Test #365: Plans1D.1DFFT_batched_2DView<std::pair<float, Kokkos::LayoutLeft>> ..................................................................   Passed   53.09 sec
        Start 368: Plans1D.1DFFT_batched_2DView<std::pair<float, Kokkos::LayoutRight>>
366/555 Test #367: Plans1D.1DFFT_1DView<std::pair<float, Kokkos::LayoutRight>> .........................................................................   Passed   30.11 sec
        Start 369: Plans1D.1DFFT_batched_3DView<std::pair<float, Kokkos::LayoutRight>>
367/555 Test #366: Plans1D.1DFFT_batched_3DView<std::pair<float, Kokkos::LayoutLeft>> ..................................................................   Passed   77.29 sec
        Start 370: Plans1D.1DFFT_1DView<std::pair<double, Kokkos::LayoutLeft>>
368/555 Test #368: Plans1D.1DFFT_batched_2DView<std::pair<float, Kokkos::LayoutRight>> .................................................................   Passed   51.60 sec
        Start 371: Plans1D.1DFFT_batched_2DView<std::pair<double, Kokkos::LayoutLeft>>
369/555 Test #370: Plans1D.1DFFT_1DView<std::pair<double, Kokkos::LayoutLeft>> .........................................................................   Passed   30.04 sec
        Start 372: Plans1D.1DFFT_batched_3DView<std::pair<double, Kokkos::LayoutLeft>>
370/555 Test #369: Plans1D.1DFFT_batched_3DView<std::pair<float, Kokkos::LayoutRight>> .................................................................   Passed   76.42 sec
        Start 373: Plans1D.1DFFT_1DView<std::pair<double, Kokkos::LayoutRight>>
371/555 Test #371: Plans1D.1DFFT_batched_2DView<std::pair<double, Kokkos::LayoutLeft>> .................................................................   Passed   51.79 sec
        Start 374: Plans1D.1DFFT_batched_2DView<std::pair<double, Kokkos::LayoutRight>>
372/555 Test #373: Plans1D.1DFFT_1DView<std::pair<double, Kokkos::LayoutRight>> ........................................................................   Passed   30.27 sec
        Start 375: Plans1D.1DFFT_batched_3DView<std::pair<double, Kokkos::LayoutRight>>
373/555 Test #372: Plans1D.1DFFT_batched_3DView<std::pair<double, Kokkos::LayoutLeft>> .................................................................   Passed   77.04 sec
        Start 376: Plans2D.2DFFT_2DView<std::pair<float, Kokkos::LayoutLeft>>
374/555 Test #374: Plans1D.1DFFT_batched_2DView<std::pair<double, Kokkos::LayoutRight>> ................................................................   Passed   56.46 sec
        Start 377: Plans2D.2DFFT_3DView<std::pair<float, Kokkos::LayoutLeft>>
375/555 Test #376: Plans2D.2DFFT_2DView<std::pair<float, Kokkos::LayoutLeft>> ..........................................................................   Passed   45.94 sec
        Start 378: Plans2D.2DFFT_2DView<std::pair<float, Kokkos::LayoutRight>>
376/555 Test #375: Plans1D.1DFFT_batched_3DView<std::pair<double, Kokkos::LayoutRight>> ................................................................   Passed   96.56 sec
        Start 379: Plans2D.2DFFT_3DView<std::pair<float, Kokkos::LayoutRight>>
377/555 Test #378: Plans2D.2DFFT_2DView<std::pair<float, Kokkos::LayoutRight>> .........................................................................   Passed   49.50 sec
        Start 380: Plans2D.2DFFT_2DView<std::pair<double, Kokkos::LayoutLeft>>
378/555 Test #377: Plans2D.2DFFT_3DView<std::pair<float, Kokkos::LayoutLeft>> ..........................................................................   Passed   71.15 sec
        Start 381: Plans2D.2DFFT_3DView<std::pair<double, Kokkos::LayoutLeft>>
379/555 Test #380: Plans2D.2DFFT_2DView<std::pair<double, Kokkos::LayoutLeft>> .........................................................................   Passed   51.53 sec
        Start 382: Plans2D.2DFFT_2DView<std::pair<double, Kokkos::LayoutRight>>
380/555 Test #379: Plans2D.2DFFT_3DView<std::pair<float, Kokkos::LayoutRight>> .........................................................................   Passed   74.28 sec
        Start 383: Plans2D.2DFFT_3DView<std::pair<double, Kokkos::LayoutRight>>
381/555 Test #381: Plans2D.2DFFT_3DView<std::pair<double, Kokkos::LayoutLeft>> .........................................................................   Passed   73.48 sec
        Start 384: Plans3D.3DFFT_3DView<std::pair<float, Kokkos::LayoutLeft>>
382/555 Test #382: Plans2D.2DFFT_2DView<std::pair<double, Kokkos::LayoutRight>> ........................................................................   Passed   50.42 sec
        Start 385: Plans3D.3DFFT_3DView<std::pair<float, Kokkos::LayoutRight>>
383/555 Test #383: Plans2D.2DFFT_3DView<std::pair<double, Kokkos::LayoutRight>> ........................................................................   Passed   74.05 sec
        Start 386: Plans3D.3DFFT_3DView<std::pair<double, Kokkos::LayoutLeft>>
384/555 Test #384: Plans3D.3DFFT_3DView<std::pair<float, Kokkos::LayoutLeft>> ..........................................................................   Passed  111.32 sec
        Start 387: Plans3D.3DFFT_3DView<std::pair<double, Kokkos::LayoutRight>>
385/555 Test #385: Plans3D.3DFFT_3DView<std::pair<float, Kokkos::LayoutRight>> .........................................................................   Passed  102.34 sec
        Start 388: FFT1D.Identity_1DView<std::pair<float, Kokkos::LayoutLeft>>
386/555 Test #386: Plans3D.3DFFT_3DView<std::pair<double, Kokkos::LayoutLeft>> .........................................................................   Passed  101.65 sec
387/555 Test #387: Plans3D.3DFFT_3DView<std::pair<double, Kokkos::LayoutRight>> ........................................................................   Passed   88.39 sec
        Start 390: FFT1D.Identity_1DView_reuse_plans<std::pair<float, Kokkos::LayoutLeft>>
388/555 Test #388: FFT1D.Identity_1DView<std::pair<float, Kokkos::LayoutLeft>> .........................................................................   Passed  629.25 sec
        Start 391: FFT1D.FFT_1DView<std::pair<float, Kokkos::LayoutLeft>>
389/555 Test #391: FFT1D.FFT_1DView<std::pair<float, Kokkos::LayoutLeft>> ..............................................................................   Passed   17.22 sec
        Start 392: FFT1D.IFFT_1DView<std::pair<float, Kokkos::LayoutLeft>>
390/555 Test #392: FFT1D.IFFT_1DView<std::pair<float, Kokkos::LayoutLeft>> .............................................................................   Passed   16.95 sec
        Start 393: FFT1D.HFFT_1DView<std::pair<float, Kokkos::LayoutLeft>>
391/555 Test #393: FFT1D.HFFT_1DView<std::pair<float, Kokkos::LayoutLeft>> .............................................................................   Passed   23.18 sec
        Start 394: FFT1D.IHFFT_1DView<std::pair<float, Kokkos::LayoutLeft>>
392/555 Test #390: FFT1D.Identity_1DView_reuse_plans<std::pair<float, Kokkos::LayoutLeft>> .............................................................   Passed  618.14 sec
        Start 395: FFT1D.FFT_1DView_shape<std::pair<float, Kokkos::LayoutLeft>>
393/555 Test #394: FFT1D.IHFFT_1DView<std::pair<float, Kokkos::LayoutLeft>> ............................................................................   Passed   15.63 sec
        Start 396: FFT1D.FFT_batched_2DView<std::pair<float, Kokkos::LayoutLeft>>
394/555 Test #389: FFT1D.Identity_1DView_inplace<std::pair<float, Kokkos::LayoutLeft>> .................................................................   Passed  672.45 sec
        Start 397: FFT1D.FFT_batched_3DView<std::pair<float, Kokkos::LayoutLeft>>
395/555 Test #395: FFT1D.FFT_1DView_shape<std::pair<float, Kokkos::LayoutLeft>> ........................................................................   Passed   41.80 sec
        Start 398: FFT1D.FFT_batched_4DView<std::pair<float, Kokkos::LayoutLeft>>
396/555 Test #396: FFT1D.FFT_batched_2DView<std::pair<float, Kokkos::LayoutLeft>> ......................................................................   Passed   50.43 sec

This was run using rocm 6.1.2.

[yasahi-hpc]

If you set the environment variables in the following manner, tests would be executed quite rapidly

export ROCFFT_RTC_SYS_CACHE_PATH=/opt/rocm/lib/rocfft/rocfft_kernel_cache.db
export ROCFFT_RTC_CACHE_PATH=./rocfft_user_cache.db
srun --ntasks-per-node 1 --cpus-per-task 1 --threads-per-core 1 --gpu-bind closest ctest --test-dir build --output-on-failure