x * y * z * 3^3 / ppn can exceed 2^31 on modern machines

[rolfriesen]

Running on a 192 GB dual socket machine. Using the MPI + OMP version in miniFE_openmp_opt
export OMP_NUM_THREADS=11
mpirun -n 4 -ppn 4 ./miniFE.x nx=682 ny=682 nz=682
throws an exception because nrows_max in CSRMatrix.hpp turns negative due to int overflow.
packed_cols.reserve(nrows_max); doesn't like negative numbers ;-)

mpirun -n 4 -ppn 4 ./miniFE.x nx=680 ny=680 nz=680 # works

Unfortunately making MINIFE_GLOBAL_ORDINAL a long is not sufficient to address the issue.

[maherou]

Hi Rolf,

C++ long is usually 32 bits. C++ long long is guaranteed to be at least 64 bits. Try setting MINIFE_GLOBAL_ORDINAL to long long and see if it helps.

[maherou]

Also, if your local problem ends up being > 2.1B in size, you can change MINIFE_LOCAL_ORDINAL to long long also.

[rolfriesen]

On Wed May 1, 2019 06:36:52, Mike Heroux wrote: Mike Also, if your local problem ends up being > 2.1B in size, you can change MINIFE_LOCAL_ORDINAL to long long also. Mike Hi Mike, I'm using the Intel Parallel Studio XE 2018 Update 3 for Linux compilers. With that, long int and long long are both 8 bytes wide. Unfortunately, changing MINIFE_LOCAL_ORDINAL to long long (or long int), causes compile errors because the function signature for init_matrix() no longer matches. To fix this, some hard coded ints in miniFE need to be changed to MINIFE_LOCAL_ORDINAL. I did create a miniFE version where I chased down all those instances and made it work. However, it it now consumes much more memory and runs 20% slower! I may have been a little overzealous with my int to MINIFE_LOCAL_ORDINAL conversion ;-) I wont have time in the next couple of weeks, but there might be a way to come up with a better fix by applying the substitution more judiciously and use type casts where appropriate. Thanks, Rolf +++-+--+----+-------+------------+--------------------+------------------------ Rolf Riesen, Ph.D. Email: rolf.riesen@intel.com Software Architect Phone: +1 (503) 613-5514 Extreme-scale Software System Pathfinding Mobile: +1 (505) 363-6871 Outlook users: Turn off "extra line break removal" in File > Options > Mail > Message Format