Gang Plugin Blocks GPU Resource Reclaim from Reclaimable Workloads

[dmildh-absci]

Description

The gang plugin seems to prevent the reclaim action from reclaiming GPU resources from reclaimable workloads, blocking jobs from obtaining necessary GPU resources. This issue specifically affects GPU resources while CPU resource reclaim continues to work normally.

Environment
Volcano Version: 1.12.1
Kubernetes Version: 1.31
Resource Types Affected: nvidia.com/gpu
Resource Types Working: cpu, memory

Problem Description
The gang plugin's MinAvailable protection logic interferes with the reclaim action, preventing GPU resources from being reclaimed from lower-priority reclaimable workloads when higher-priority jobs need those resources.

Current Scheduler Configuration

    actions: "enqueue, allocate, reclaim, backfill"
    tiers:
    - plugins:
      - name: priority
      - name: gang
      - name: conformance
    - plugins:
      - name: predicates
      - name: capacity
      - name: drf
      - name: nodeorder
      - name: binpack
        arguments:
          binpack.weight: 10
          binpack.cpu: 2
          binpack.memory: 2
          binpack.resources: "nvidia.com/gpu"
          binpack.resources.nvidia.com/gpu: 8
          binpack.score.gpu.weight: 10

Logs of the Gang plugin blocking reclaim operation:

gang.go:97] Can not preempt task <default/reclaimable-gpu-1> because job podgroup-xxx ready num(1) <= MinAvailable(1) for gang-scheduling
gang.go:97] Can not preempt task <default/reclaimable-gpu-2> because job podgroup-xxx ready num(1) <= MinAvailable(1) for gang-scheduling

Result: victims=[] (no GPU resources can be reclaimed)

Steps to reproduce the issue

On a cluster which a single gpu node setup queues like so

---
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: proj2-gpu
spec:
  weight: 1
  reclaimable: false
  capability:
    nvidia.com/gpu: 1
  deserved:
    nvidia.com/gpu: 1
---
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: reclaimable-gpu
spec:
  weight: 1
  reclaimable: true
  capability:
    nvidia.com/gpu: 1

Run a test pod to consume the gpu

---
apiVersion: v1
kind: Pod
metadata:
  name: reclaimable-gpu-1
  annotations:
    scheduling.volcano.sh/queue-name: reclaimable-gpu
spec:
  schedulerName: volcano
  containers:
  - name: proj-gpu-workload
    image: nginx:1.14.2
    command: ["sleep"]
    args: ["infinity"]
    resources:
      requests:
        nvidia.com/gpu: 1
      limits:
        nvidia.com/gpu: 1
  tolerations:
  - key: "nvidia.com/gpu"
    operator: "Equal"
    value: "present"
    effect: "NoSchedule"

Attempt to reclaim it

apiVersion: v1
kind: Pod
metadata:
  name: proj1-gpu
  annotations:
    scheduling.volcano.sh/queue-name: proj1-gpu
spec:
  schedulerName: volcano
  containers:
  - name: proj-gpu-workload
    image: nginx:1.14.2
    command: ["sleep"]
    args: ["infinity"]
    resources:
      requests:
        nvidia.com/gpu: 1
      limits:
        nvidia.com/gpu: 1
  tolerations:
  - key: "nvidia.com/gpu"
    operator: "Equal"
    value: "present"
    effect: "NoSchedule"

Describe the results you received and expected

The reclaim pod is not reclaimed when using a gpu based resource. If you do the same testing with cpu/memory resources it works.

What version of Volcano are you using?

1.12.1

Any other relevant information

No response

[my-zzm]

I meet the same question

[lowang-bh]

you can disable it in config

[hajnalmt]

This is a quite common issue. Disable the preemtable action in the gang plugin.
Your config modified:

actions: "enqueue, allocate, reclaim, backfill"
tiers:
- plugins:
  - name: priority
  - name: capacity
  - name: gang
    enablePreemptable: false  # This allows preemption without gang constraints
  - name: conformance
- plugins:
  - name: predicates
  - name: drf
  - name: nodeorder
  - name: binpack

[dmildh-absci]

How is the preemtable action related here? I thought that is completely separate than reclaim. Also it looks like I posted the incorrect config above we are using the following: (going to update the original here as well)

    actions: "enqueue, allocate, reclaim, backfill"
    tiers:
    - plugins:
      - name: priority
      - name: gang
      - name: conformance
    - plugins:
      - name: predicates
      - name: capacity
      - name: drf
      - name: nodeorder
      - name: binpack
        arguments:
          binpack.weight: 10
          binpack.cpu: 2
          binpack.memory: 2
          binpack.resources: "nvidia.com/gpu"
          binpack.resources.nvidia.com/gpu: 8
          binpack.score.gpu.weight: 10

[dmildh-absci]

After much testing I believe I now have a working configuration. The import part here is that you have some value defined in your queues for cpu and memory as well. Doesn't really matter what they are if you only care about gpu count like we do.

So it would need to look like this for example. Notice the overly large cpu and memory figures. Since we are presizing pods based on the amount of gpus those numbers are not as important to us.

---
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: proj2-gpu
spec:
  weight: 1
  reclaimable: false
  capability:
    cpu: 1000
    mem: 100Ti
    nvidia.com/gpu: 1
  deserved:
    cpu: 1000
    mem: 100Ti
    nvidia.com/gpu: 1
---
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: reclaimable-gpu
spec:
  weight: 1
  reclaimable: true
  capability:
    cpu: 1000
    mem: 100Ti
    nvidia.com/gpu: 1

This requirement should at least be added into the docs somewhere if it's not already

[dmildh-absci]

Sorry looks like it's still busted. reclaim only works if you use podgroup only and not vcjobs which seems to trigger something in the gang plugin itself.

[JesseStutler]

How is the preemtable action related here? I thought that is completely separate than reclaim. Also it looks like I posted the incorrect config above we are using the following: (going to update the original here as well)

    actions: "enqueue, allocate, reclaim, backfill"
    tiers:
    - plugins:
      - name: priority
      - name: gang
      - name: conformance
    - plugins:
      - name: predicates
      - name: capacity
      - name: drf
      - name: nodeorder
      - name: binpack
        arguments:
          binpack.weight: 10
          binpack.cpu: 2
          binpack.memory: 2
          binpack.resources: "nvidia.com/gpu"
          binpack.resources.nvidia.com/gpu: 8
          binpack.score.gpu.weight: 10

Actually it's "enableReclaimable".
So your scheduler config should be:

    actions: "enqueue, allocate, reclaim, backfill"
    tiers:
    - plugins:
      - name: priority
      - name: gang
         enableReclaimable: false # This allows reclaim below minAvailable
      - name: conformance
    - plugins:
      - name: predicates
      - name: capacity
      - name: drf
      - name: nodeorder
      - name: binpack
        arguments:
          binpack.weight: 10
          binpack.cpu: 2
          binpack.memory: 2
          binpack.resources: "nvidia.com/gpu"
          binpack.resources.nvidia.com/gpu: 8
          binpack.score.gpu.weight: 10

@dmildh-absci

[JesseStutler]

Sorry looks like it's still busted. reclaim only works if you use podgroup only and not vcjobs which seems to trigger something in the gang plugin itself.

Sorry, what does this mean? VC-Controller will create podgroups for the vcjob, and also orphan pod. VC-Scheduler only schedule podgroups

[JesseStutler]

After much testing I believe I now have a working configuration. The import part here is that you have some value defined in your queues for cpu and memory as well. Doesn't really matter what they are if you only care about gpu count like we do.

So it would need to look like this for example. Notice the overly large cpu and memory figures. Since we are presizing pods based on the amount of gpus those numbers are not as important to us.

---
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: proj2-gpu
spec:
  weight: 1
  reclaimable: false
  capability:
    cpu: 1000
    mem: 100Ti
    nvidia.com/gpu: 1
  deserved:
    cpu: 1000
    mem: 100Ti
    nvidia.com/gpu: 1
---
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: reclaimable-gpu
spec:
  weight: 1
  reclaimable: true
  capability:
    cpu: 1000
    mem: 100Ti
    nvidia.com/gpu: 1

This requirement should at least be added into the docs somewhere if it's not already

Is the problem solved now? But I see that the problem description is still because the gang plugin blocks reclaim, is there any other problem we need to fix?