Segmentation Fault 11 with PDLIB

[seimurss]

Dear Community,

The error is:

[n047:386467:0:386467] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x2b34b1ccde90)
[n076:302046:0:302046] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x2b4992bcfcb8)
......
==== backtrace (tid:  47156) ====
 0 0x000000000057dde5 w3ref1md_mp_w3sref_()  ???:0
 1 0x000000000055cfe9 w3srcemd_mp_w3srce_()  ???:0
 2 0x00000000004a1843 w3wavemd_mp_w3wave_()  ???:0
 3 0x0000000000417132 MAIN__()  ???:0
 4 0x000000000040ff12 main()  ???:0
 5 0x00000000000223d5 __libc_start_main()  ???:0
 6 0x000000000040fe29 _start()  ???:0

I am facing the above segmentation fault (11) while running my scenario on 216 cores (various core nums were tried, from 1 to 512) with the PDLIB switch. Although, when i cut my domain, or significantly simplify the grid, into small chunks and execute them individually they works fine, as long as the partitions are relatively simple.

It seems that the problem lays in the domain decomposition or partitioning. Perhaps parMetis has something to do with this that is used by the PDLIB switch or overall configuration (including the libraries' versions that i load) along with the grid itself have some peculiarities that lead to this fault.

There are other scenarios at different regions (grid) where the same configurations work fine without the need of portioning into small chunks, thus the problem might be with the grid itself, however, narrowing down the grid and having left with almost 'perfect' triangles, i don't seem to spot an issue of any sorts.

I have tried to refine the grid removing the complicated joints or eliminating extremely skewed triangles and so on. There are also a few checks that i run over the final grid to ensure that there are no unconnected triangles/nodes/elements, no extreme angles or wrong connections, no quads or other figures.

If anyone has had similar experience or has some ideas on how to approach to problem, could you please help me out?

Thank you!

Some configuration details are below:

• Unstructured grid made of triangles of around ~170k nodes and ~350k elements

• ww3 model v607.1 is compiled using mpifortran.

• The boundary forcing is turned off, only wind is on.

• The list of switches: F90 NOGRB NC4 DIST MPI PR3 UQ FLX4 LN1 ST6 STAB0 NL1 BT4 DB1 MLIM TR1 BS0 IC0 IS0 REF1 XX0 WNT1 WNX1 CRT1 CRX1 TRKNC PDLIB O0 O1 O2 O4 O5 O6 O7

• The list of libraries used during the compilation of ww3 and the execution of the scenario: udunits/2.2.26, szip/2.1.1, curl/7.66.0, gcc_9.1.0, 20.1.217, hdf5/1.12.0, netcdf/C_4.7.4-F_4.5.3_CXX_4.3.1, ncview/2.1.8, cmake/3.17.3 , impi20.1, impi20.1/19.7.217, impi20.1/parallel-netcdf/1.12.1, impi20.1/netcdf/C_4.7.4-F_4.5.3_CXX_4.3.1, anaconda/3.9

[aliabdolali]

Change REF1 to REF0 and give it a try again.

[seimurss]

@aliabdolali It seems to have worked, but is there any other way we can work around the issue without turning off the reflection? Or maybe this, having worked, hints at certain problems within the grid that you might know about? Thank you!

[aliabdolali]

have you tried the latest version? from develop? not the public release. I think we've fixed it.

[seimurss]

I am trying it now, although facing many issues along the way as there are quite many changes from the latest stable version.
If I may ask, could you please direct me to the module(s) that's been changed to solve the above segmentation fault issue, so that i could look at and better understand the problem myself and perhaps try to implement just that change while leaving the rest untouched?

Thanks a lot!

[aronroland]

let me know how it goes ...