MPICH high memory footprint

[aditya-nishtala]

We ran a simple hello world mpich program where each rank prints the rank id + hostname its running on.
The program allocates no memory at all, all of the memory allocation comes from whatever MPICH is doing.

We scaled the from 32 nodes to 768 nodes and measured how much memory is being consumed.
MPICH commit tag is 204f8cd
This is happening on Aurora

Memory Consumption is equivalent whether using DDR or HBM. Below data is measured on DDR.

	Max DDR utilization (GB)
Node count	mpich/ opt/ develop-git.204f8cd
32	22.53
64	24.58
128	28.16
256	35.33
512	50.18
768	68.10

[nsdhaman]

Note that above data is with PPN 96. The reported memory footprint values are in GB per socket. There is linear increase in memory overhead and it is persisting through entire program execution.

[hzhou]

@aditya-nishtala Could you retry the experiment using a debug-build and enable MPIR_CVAR_DEBUG_SUMMARY=1 and then post one of the log? That may help identify whether the memory is allocated by MPICH or by one of its dependent libraries.

[hzhou]

Taking the difference, the memory increase are roughly linear to the number of nodes, ~55-65 MB/Node. @aditya-nishtala How many PPN (process per node)?

[nsdhaman]

This is with PPN 96.

[hzhou]

Thanks @nsdhaman . So that is roughly 6KB per connection.

[hzhou]

Okay, I think the issue is we are allocating too much address table prepared for all possible connections. If we assume no application will use multiple VCI, we could configure with --with-ch4-max-vcis=1, that will cut down the memory growth by 1/64.

For more appropriate fix, we could change the av table accommodate multi-VCI/NIC entries dynamically rather than statically. I probably can implement something like that.