[Feature] Support Volcano Network Topology Aware Scheduling for kuberay #4105

mtian29 · 2025-10-01T00:45:58Z

Why are these changes needed?

Support for Volcano Network Topology Aware Scheduling

Close issue. #3641

Test

Build a image and use this image for kuberay operator

Apply a raycluster with labels

apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: kwok-raycluster-h100-q21-low
  labels:
    ray.io/scheduler-name: volcano 
    volcano.sh/queue-name: queue2 
    ray.io/priority-class-name: ml-tier1 
    volcano.sh/network-topology-mode: hard # <----
    volcano.sh/network-topology-highest-tier-allowed: "1" # <-----
.....

The volcano podgroup has the field injected

❯ kg pg -oyaml
apiVersion: v1
items:
- apiVersion: scheduling.volcano.sh/v1beta1
  kind: PodGroup
  metadata:
    annotations:
      volcano.sh/job-allocated-hypernode: hypernode-ad1
    creationTimestamp: "2025-09-30T22:40:12Z"
    generation: 3
    name: ray-kwok-raycluster-h100-q21-low-pg
    namespace: kuberay
.....
  spec:
    minMember: 10
    minResources:
      cpu: "80"
      memory: 80Gi
      nvidia.com/h100: "80"
    networkTopology:
      highestTierAllowed: 1 <--- 
      mode: hard <-----

Some other combinations of labels and result.

volcano.sh/network-topology-mode: soft
volcano.sh/network-topology-highest-tier-allowed: "2"

networkTopology:
  highestTierAllowed: 2
  mode: soft

—
volcano.sh/network-topology-mode: hard
volcano.sh/network-topology-highest-tier-allowed: "2"

networkTopology:
  highestTierAllowed: 2
  mode: hard

—
volcano.sh/network-topology-mode: hard

networkTopology:
  highestTierAllowed: 1 <— default
  mode: hard

——
volcano.sh/network-topology-mode: soft

networkTopology:
  highestTierAllowed: 1 <— default
  mode: soft

——

no label => no network topology

spec:
minMember: 20
minResources:
cpu: "160"
memory: 160Gi
nvidia.com/h100: "160"
priorityClassName: ml-tier1
queue: queue2

—-

volcano.sh/network-topology-mode: soft
volcano.sh/network-topology-highest-tier-allowed: "abc"

"PodGroup.Error":"failed to convert volcano.sh/network-topology-highest-tier-allowed label to int: strconv.Atoi: parsing "abc": invalid syntax for podgroup ray-kwok-raycluster-h100-q2-soft-topology2-pg in namespace kuberay

Related issue number

Closes #3641

Checks

I've made sure the tests are passing.
Testing Strategy
- Unit tests
- Manual tests
- This PR is not tested :(

fscnick · 2025-10-01T14:26:45Z

ray-operator/controllers/ray/batchscheduler/volcano/volcano_scheduler.go

+	mode, modeOk := app.ObjectMeta.Labels[NetworkTopologyModeLabelKey]
+	highestTier, tierOk := app.ObjectMeta.Labels[NetworkTopologyHighestTierAllowedLabelKey]
+	if modeOk && tierOk {
+		highestTierInt, err := strconv.Atoi(highestTier)


Should we handle this error so that the user will get a better understanding of what happened?

we should. Changed my PR. Thanks for reviewing.

fscnick · 2025-10-01T14:32:15Z

ray-operator/controllers/ray/batchscheduler/volcano/volcano_scheduler.go


+	mode, modeOk := app.ObjectMeta.Labels[NetworkTopologyModeLabelKey]
+	highestTier, tierOk := app.ObjectMeta.Labels[NetworkTopologyHighestTierAllowedLabelKey]
+	if modeOk && tierOk {


According to Network Topology Aware Scheduling Policy, the highestTierAllowed is not required if mode is soft. If the highestTierAllowed is not set with soft mode, the NetworkTopologySpec would not be set.

this is a good catch.
Changed my PR.

win5923

I think we should synchronize the NetworkTopology field for PodGroup when updating an existing RayCluster labels. WDYT?

mtian29 · 2025-10-01T22:50:10Z

I think we should synchronize the NetworkTopology field for PodGroup when updating an existing RayCluster labels. WDYT?

Thanks. @win5923 Good point.

Where should we do this for update? I don't see a place to do it.
But even we can, we don't need to do it. Network topology is for scheduling, once a pod is bind to a node and initializing, changing labels/spec/network topology won't move the pod to a different node.

win5923 · 2025-10-02T16:39:38Z

Thanks. @win5923 Good point.

Where should we do this for update? I don't see a place to do it.
But even we can, we don't need to do it. Network topology is for scheduling, once a pod is bind to a node and initializing, changing labels/spec/network topology won't move the pod to a different node.

We can do this in syncPodGroup, but I think you’re right.
This is only for scheduling. If users want to change the topology settings, they should recreate the RayCluster.

win5923

Thanks for the contribution, the changes look good to me.
But i think we should wait until #3972
is merged, as it includes the interface changes that might impact this implementation.

mtian29 · 2025-10-03T18:49:51Z

Thanks for the contribution, the changes look good to me. But i think we should wait until #3972 is merged, as it includes the interface changes that might impact this implementation.

Thanks @win5923
The only impacted part should be app.ObjectMeta.Labels[xxxxx] => owner.GetLabels()[xxxx]

Do you know when is the next release of the kuberay operator. Can my change be included?

win5923 · 2025-10-03T18:59:20Z

Do you know when is the next release of the kuberay operator. Can my change be included?

Nov. 1, 2025 (Branch Cut: Oct. 10).
Ref: https://docs.google.com/document/d/1rdXniNitHCNTGfyvvMPdMkp1cDnQtjOmfqNiWuvrS9A/edit?tab=t.0#heading=h.ctb1p12e6p4u

Yes, I believe this PR can be included in v1.5, as the changes are relatively minor and should be safe to merge before the cut-off.

troychiu · 2025-10-03T23:39:48Z

Sorry for the late reply. I'll review this PR asap.

troychiu

Hi since #3972 has been merged, do you mind resolving the conflicts? Thank you!

mtian29 · 2025-10-10T17:15:14Z

Hi since #3972 has been merged, do you mind resolving the conflicts? Thank you!

Thanks @troychiu . Rebased.

troychiu

Thank you!

ray-operator/controllers/ray/batchscheduler/volcano/volcano_scheduler_test.go

mtian29 · 2025-10-11T00:03:37Z

Thank you!

@troychiu Thanks. I changed the tests following your suggestions and also fixed the lint errors.

troychiu · 2025-10-11T01:01:11Z

cc @Future-Outlier @rueian

mtian29 requested review from MortalHappiness, andrewsykim, kevin85421 and rueian as code owners October 1, 2025 00:45

mtian29 changed the title ~~Support for Volcano Network Topology Aware Scheduling~~ [Feature] Support for Volcano Network Topology Aware Scheduling Oct 1, 2025

mtian29 changed the title ~~[Feature] Support for Volcano Network Topology Aware Scheduling~~ [Feature] Support for Volcano Network Topology Aware Scheduling for kuberay Oct 1, 2025

mtian29 mentioned this pull request Oct 1, 2025

[Feature] Support for Volcano Network Topology Aware Scheduling #3641

Open

2 tasks

Future-Outlier assigned Future-Outlier and win5923 and unassigned Future-Outlier Oct 1, 2025

fscnick reviewed Oct 1, 2025

View reviewed changes

win5923 reviewed Oct 1, 2025

View reviewed changes

mtian29 changed the title ~~[Feature] Support for Volcano Network Topology Aware Scheduling for kuberay~~ [Feature] Support Volcano Network Topology Aware Scheduling for kuberay Oct 1, 2025

win5923 approved these changes Oct 3, 2025

View reviewed changes

troychiu reviewed Oct 10, 2025

View reviewed changes

Support Volcano Network Topology Aware Scheduling for kuberay

f9cb394

mtian29 force-pushed the mtian/volcano-topology branch from 8599979 to f9cb394 Compare October 10, 2025 16:49

troychiu approved these changes Oct 10, 2025

View reviewed changes

ray-operator/controllers/ray/batchscheduler/volcano/volcano_scheduler_test.go Outdated Show resolved Hide resolved

Mingyuan Tian added 2 commits October 10, 2025 23:37

Address comments for tests

e575f91

Fix precommit lint errors.

cb97330

rueian approved these changes Oct 11, 2025

View reviewed changes

rueian merged commit f22a75a into ray-project:master Oct 11, 2025
27 checks passed

[Feature] Support Volcano Network Topology Aware Scheduling for kuberay #4105

[Feature] Support Volcano Network Topology Aware Scheduling for kuberay #4105

Uh oh!

Conversation

mtian29 commented Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Test

Related issue number

Checks

Uh oh!

fscnick Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

mtian29 Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

fscnick Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

mtian29 Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

win5923 left a comment

Choose a reason for hiding this comment

Uh oh!

mtian29 commented Oct 1, 2025

Uh oh!

win5923 commented Oct 2, 2025

Uh oh!

win5923 left a comment

Choose a reason for hiding this comment

Uh oh!

mtian29 commented Oct 3, 2025

Uh oh!

win5923 commented Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

troychiu commented Oct 3, 2025

Uh oh!

troychiu left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mtian29 commented Oct 10, 2025

Uh oh!

troychiu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mtian29 commented Oct 11, 2025

Uh oh!

troychiu commented Oct 11, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

mtian29 commented Oct 1, 2025 •

edited

Loading

win5923 commented Oct 3, 2025 •

edited

Loading

troychiu left a comment •

edited

Loading