[Trainer][torch-distributed/DDP] Communication bottleneck between pods during multi-node training — guidance for optimization and diagnosis

[luigicollesi]

Summary
Running PyTorch DDP on Kubeflow Trainer (torch-distributed runtime) shows high communication overhead during multi-node training. Link speed is 2.5 Gbit/s, but effective pod-to-pod throughput during training plateaus around ~1.4 Gbit/s, limiting scaling.

Measured throughput
Using an internal measurement script that times transfers of varying message sizes (P2P vs. collective) for ~1 minute, inter-node bandwidth is ~1.8 Gbit/s with a few machines and ~1.4 Gbit/s when using all 7 nodes.

What I observe

Collective ops dominate step time beyond one node.

Sub-linear speedup as total GPUs/nodes increase.

Ask

Kubeflow-specific best practices to improve NCCL communication on Kubernetes.

Recommended Trainer/runtime settings or pod/network tweaks (and any example manifests) for higher throughput.

Known CNI/overlay pitfalls that could explain the gap between link speed (2.5 Gbit/s) and effective throughput (1.4–1.5 Gbit/s).

Environment

Kubernetes version:
Client Version: v1.33.2
Kustomize Version: v5.6.0
Server Version: v1.33.2

Kubeflow Trainer version:

Kubeflow Python SDK version:
Name: kubeflow
Version: 0.1.0
Summary: Kubeflow Python SDK to manage ML workloads and to interact with Kubeflow APIs.
Home-page: https://github.com/kubeflow/sdk
Author:
Author-email: The Kubeflow Authors kubeflow-discuss@googlegroups.com
License-Expression: Apache-2.0
Location: /opt/conda/lib/python3.11/site-packages
Requires: kubeflow-trainer-api, kubernetes, pydantic

[andreyvelich]

Thanks for creating this @luigicollesi!

This is great idea, do you know if there are any NCCL env configurations we can set to reduce communication overhead ?
Also, do you use NVLink in your cluster setup?

Adding best practices guide to Trainer docs might be a good idea: https://www.kubeflow.org/docs/components/trainer/user-guides/

/area performance

[luigicollesi]

Thanks for following up!

NVLink: Not configured/available in this cluster (GPUs communicate over PCIe only; no NVLink fabric).

NCCL environment variables I explicitly tested to try to improve pod-to-pod communication (no improvement):

NCCL_SOCKET_IFNAME=eth0
# Goal: pin NCCL to the pod’s primary NIC to avoid interface ambiguity and
#       keep traffic on the expected data path.

NCCL_SOCKET_FAMILY=AF_INET
# Goal: force IPv4 sockets to avoid IPv6 resolution paths and any dual-stack
#       overhead or fallbacks.

NCCL_IB_DISABLE=1
# Goal: ensure NCCL uses TCP sockets only (no RDMA transports), removing
#       any misdetection or partial IB/RoCE code paths.

NCCL_PROTO=Simple
# Goal: use the “Simple” protocol, which can be more stable for larger
#       messages and higher latencies vs. “LL/LL128” in some setups.

Outcome: I tested these variables individually and in combination, specifically to see if they would increase effective collective bandwidth / reduce step time for the Trainer pods. They did not produce any measurable improvement in communication speed or training throughput.

[andreyvelich]

@luigicollesi Do you set any label selectors/tolerations to ensure that your node are co-located ?
Also, do you have any CNI in your cluster (e.g. Calico, Istio) ? That might slow down the communication between Pods.

---

You can consider to run nccl-test for more detailed information for your pod-to-pod throughput: https://github.com/NVIDIA/nccl-tests