Why is my program hitting "Error in ADIOI_Calc_aggregator()"

[nahkbce2]

Hi,
We have a parallel application with intel MPI (which is based on MPICH) and use SLURM for our HPC. Recently some of our jobs gives this error:
"Error in ADIOI_Calc_aggregator(): rank_index(%d) >= fd->hints->cb_nodes (%d) fd_size=%lld off=%lld\n",Error in ADIOI_Calc_aggregator(): rank_index(850069981) >= fd->hints->cb_nodes (1) fd_size=8594349943 off=7305798900674814957

application called MPI_Abort(MPI_COMM_WORLD, 1) - process 23

searching this repository i could see a comment in src/mpi/romio/adio/common/ad_aggregate.c and
src/mpi/romio/adio/ad_gpfs/ad_gpfs_aggrs.c; thats its overrunning arrays.

What does that mean and how can i tackle this problem?
Thanks

[hzhou]

Could you try build the latest MPICH (https://www.mpich.org/downloads/) e.g. mpich-4.1rc1 and see if the application still crashes?

[nahkbce2]

Hi,
We cant really use MPICH directly only use the one shipped with Intel's HPC package. Is there an upgraded INTELs MPI that we could try?

[hzhou]

The error message indicates a data overflow or data corruption. We will need a reproducer or a backtrace in order to trouble-shoot.

[nahkbce2]

okay, let me try and come-up with one.