[Issue]RCCL 2.22.3-v1 alltoall with PXN will not do the message aggregation

[shanleo2024]

Hi dear developer,

We tested the RCCL 2.22.3-v1, alltoall with PXN open can not do the message aggregation. While NCCL 2.22.3 in A100, alltoall with PXN open can do the message aggregation, can you give me some comments why RCCL don't do the message aggregation?
Thank you.

[nileshnegi]

just clarifying, in RCCL, PXN is disabled by default... so are you using NCCL_PXN_DISABLE=0 to use PXN with RCCL?
also, can you shed some light on how you are measuring/checking for message aggregation?

[shanleo2024]

just clarifying, in RCCL, PXN is disabled by default... so are you using NCCL_PXN_DISABLE=0 to use PXN with RCCL? also, can you shed some light on how you are measuring/checking for message aggregation?
Hi @nileshnegi
In deed I set 'NCCL_PXN_DISABLE=0' to enable PXN manually, by checking the code: recvProxyProgress
args->nsubs is always 1, so there is no multi recv for PXN.
I has set those two variables already:
NCCL_NET_SHARED_BUFFERS=1
NCCL_NET_SHARED_COMMS=1

If I have some misunderstanding, please correct me, thank you.

[shanleo2024]

Hi @nileshnegi
I noticed NCCL_MAX_DEV_WORK_P2P_PER_BATCH is set to 1 in RCCL, while 8 in NCCL
What considerations led to setting this value to 1？
and this one:
NCCL:

#define NCCL_MAX_DEV_WORK_BATCH_BYTES 1024
#define NCCL_MAX_DEV_WORK_BATCH_COLLS (NCCL_MAX_DEV_WORK_BATCH_BYTES/sizeof(ncclDevWorkColl))
#define NCCL_MAX_DEV_WORK_P2P_PER_BATCH 8

RCCL:

#define NCCL_MAX_DEV_WORK_BATCH_BYTES 128
#define NCCL_MAX_DEV_WORK_BATCH_COLLS (NCCL_MAX_DEV_WORK_BATCH_BYTES/sizeof(ncclDevWorkColl))
#define NCCL_MAX_DEV_WORK_P2P_PER_BATCH 1

As NCCL_MAX_DEV_WORK_P2P_PER_BATCH is set to 1, the alltoall PXN will not do the message aggregation indeed.
Wish for receiving your response, thank you.