2 GPUs error invalid permissions for mapped object at address 0x7fb5591e2c00)

[ztdepztdep]

I can run with cuda backend with 1 gpu sucessfully. these two gpu both can run it smoothly. But when i tried to compile with 2 GPUs , it feeds back the error . "Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x7f69139e2e00)"

`nvidia-smi
Sat Feb 8 17:58:29 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.142 Driver Version: 550.142 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 2060 Off | 00000000:03:00.0 On | N/A |
| 30% 37C P8 10W / 172W | 646MiB / 6144MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA GeForce GTX 1070 Ti Off | 00000000:04:00.0 Off | N/A |
| 0% 41C P8 7W / 180W | 8MiB / 8192MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

`

             __    ____  _____

____ ___ / /__ / __ / /
/ __ \ / _ \ / //// // /_ \
/ / / // // ,< / , // /
// // __///||// ||/____/ v22.0.0 (no sha)

COPYRIGHT (c) 2019-2022 UCHICAGO ARGONNE, LLC

MPI tasks: 2

reading par file ...

using NEKRS_HOME: /run/media/ztdep/hpc/nekrsv22
using NEKRS_CACHE_DIR: /run/media/ztdep/hpc/nekrsv22/run/eddyPeriodic/.cache
using OCCA_CACHE_DIR: /run/media/ztdep/hpc/nekrsv22/run/eddyPeriodic/.cache/occa/

Initializing device
active occa mode: CUDA

building udf ...
[100%] Built target UDF
done (0.105907s)
skip building nekInterface (SIZE requires no update)
loading nek ...
done
loading kernels (this may take awhile) ...
loading udf kernels ... done (0.000112968s)
Ax: N=7 wordSize=64 GDOF/s=0.866043 GB/s=82.7361 GFLOPS/s=143.495 bkMode=1 kernelVer=2
Ax: N=7 wordSize=64 GDOF/s=0.896075 GB/s=85.6052 GFLOPS/s=148.471 bkMode=1 kernelVer=0
Ax: N=7 wordSize=32 GDOF/s=2.28608 GB/s=109.199 GFLOPS/s=378.783 bkMode=1 kernelVer=6
fdm: N=9 wordSize=32 GDOF/s=5.3533 GB/s=96.9322 GFLOPS/s=888.545 kernelVer=1
Ax: N=3 wordSize=64 GDOF/s=0.179699 GB/s=27.2611 GFLOPS/s=26.8351 bkMode=1 kernelVer=1
Ax: N=3 wordSize=32 GDOF/s=0.175716 GB/s=13.3284 GFLOPS/s=26.2402 bkMode=1 kernelVer=6
fdm: N=5 wordSize=32 GDOF/s=0.969798 GB/s=23.4614 GFLOPS/s=122.334 kernelVer=0
done (5.49571s)

Reading /run/media/ztdep/hpc/nekrsv22/run/eddyPeriodic/eddy.re2
reading mesh
reading boundary faces 64 for ifield 1
done :: read .re2 file 0.35E-02 sec

Running parCon ... (tol=0.2)
Running parRSB ...
parRSB finished in 0.00259047 s

reading mesh
reading boundary faces 64 for ifield 1
done :: read .re2 file 0.13E-02 sec

setup mesh topology
Right-handed check complete for 256 elements. OK.
gs_setup: 1568 unique labels shared
pairwise times (avg, min, max): 8.26945e-06 8.2263e-06 8.3126e-06
crystal router : 7.09385e-06 7.0261e-06 7.1616e-06
all reduce : 1.77916e-05 1.77066e-05 1.78765e-05
used all_to_all method: crystal router
handle bytes (avg, min, max): 534292 534292 534292
buffer bytes (avg, min, max): 50176 50176 50176
setupds time 1.4207E-02 seconds 0 8 45056 256

nElements max/min/bal: 128 128 1.00
nMessages max/min/avg: 1 1 1.00
msgSize max/min/avg: 1568 1568 1568.00
msgSizeSum max/min/avg: 1568 1568 1568.00

max multiplicity 8
done :: setup mesh topology

call usrdat
done :: usrdat

generate geometry data
done :: generate geometry data

call usrdat2
done :: usrdat2

0.0000E+00 0.0000E+00 0.0000E+00 0.0000E+00 0.0000E+00 0.0000E+00 xyz repair 1
0.0000E+00 0.0000E+00 0.0000E+00 0.0000E+00 0.0000E+00 0.0000E+00 xyz repair 2
0.0000E+00 0.0000E+00 0.0000E+00 0.0000E+00 0.0000E+00 0.0000E+00 xyz repair 3
0.0000E+00 0.0000E+00 0.0000E+00 0.0000E+00 0.0000E+00 0.0000E+00 xyz repair 4
regenerate geometry data 1
done :: regenerate geometry data 1

regenerate geometry data 1
done :: regenerate geometry data 1

verify mesh topology
0.0000000000000000 6.2831853071795862 Xrange
0.0000000000000000 6.2831853071795862 Yrange
0.0000000000000000 1.0000000000000000 Zrange
done :: verify mesh topology

mesh metrics:
GLL grid spacing min/max : 2.52E-02 2.09E-01
scaled Jacobian min/max/avg: 1.00E+00 1.00E+00 1.00E+00
aspect ratio min/max/avg: 2.55E+00 2.55E+00 2.55E+00

call usrdat3
done :: usrdat3

gridpoints unique/tot: 87808 131072
dofs vel/pr: 87808 87808
nek setup done in 1.8729E-01 s

set initial conditions
nekuic (1) for ifld 1
call nekuic for vel
xyz min 0.0000 0.0000 0.0000
uvwpt min -1.0000 -1.4120 0.0000 0.0000 0.0000
PS min 0.0000 0.0000 0.0000 0.99000E+22
xyz max 6.2832 6.2832 1.0000
uvwpt max 3.0000 2.0120 0.0000 0.0000 0.0000
PS max 0.0000 0.0000 0.0000 -0.99000E+22
done :: set initial conditions

calling nek_userchk ...
setting vx,vy,pr 0 0.0000000000000000 5.0000000000000003E-002

    0  0.0000E+00 Write checkpoint
   FILE:/run/media/ztdep/hpc/nekrsv22/run/eddyPeriodic/eddy0.f00001                                                                         

min/max: 0.0000 6.2832 0.0000 6.2832 0.0000 1.0000
min/max: -1.0000 3.0000 -1.4120 2.0120 -1.0000 3.0000
min/max: -3.5995 1.3906

    0  0.0000E+00 done :: Write checkpoint
                          file size =     369.E-05GB
                          avg data-throughput =     0.6GB/s
                          io-nodes =     2


    0  0.0000E+00 Write checkpoint
   FILE:/run/media/ztdep/hpc/nekrsv22/run/eddyPeriodic/erreddy0.f00001                                                                      

min/max: 0.0000 6.2832 0.0000 6.2832 0.0000 1.0000
min/max: 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
min/max: 0.0000 0.0000

    0  0.0000E+00 done :: Write checkpoint
                          file size =     369.E-05GB
                          avg data-throughput =     0.7GB/s
                          io-nodes =     2

0  0.000000E+00  0.000000E+00  3.000000E+00  3.000000E+00  1.000000E+00  X err
0  0.000000E+00  0.000000E+00  2.011970E+00  2.011970E+00  3.000000E-01  Y err
0  0.000000E+00  0.000000E+00  3.599496E+00  3.599496E+00                P err

generating t-mesh ...
loading mesh from nek ... NboundaryIDs: 0, NboundaryFaces: 1536 done (6.4979e-05s)
N: 7, Nq: 8, cubNq: 11
computing geometric factors ... J [0.0192766,0.0192766] done (0.0678739s)
meshParallelGatherScatterSetup N=7
timing gs modes: 2.13e-05s 1.04e-04s 1.06e-04s 1.02e-04s

[stgeke]

Your MPI installation seems to lack GPU support. You can run nekRS without using the env-var NEKRS_GPU_MPI=0.

[ztdepztdep]

Your MPI installation seems to lack GPU support. You can run nekRS without using the env-var NEKRS_GPU_MPI=0.

thanks. it works. but the flops value decreased from 4E10 to 7E9 when I use 2 GPUs. what had happened?

[stgeke]

What were your expectations when running on an RTX 2060 and a GTX 1070? Utilizing more than one GPU requires MPI, which introduces overhead. The actual impact varies based on factors such as local problem size, interconnect, and other system characteristics.