Spark Applications submitted by Spark Operator cannot reclaim resource between queues

[Jylx]

Description

When using volcano hierarchical queues, Spark Applications submitted by Spark Operator cannot reclaim resource between queues.

I want to confirm if it's a configuration problem or if it triggered a bug?
I need help, any suggestions would be greatly appreciated.
Thank you very much.

Steps to reproduce the issue

1. volcano scheduler configmap is

kubectl get configmaps -n volcano-system volcano-scheduler-configmap -o yaml

apiVersion: v1
data:
  volcano-scheduler.conf: |
    actions: "enqueue, allocate, preempt, reclaim, backfill" 
    tiers:
    - plugins:
      - name: priority
      - name: gang
        enablePreemptable: false
      - name: conformance
    - plugins:
      - name: overcommit
      - name: drf
        enablePreemptable: false
      - name: predicates
      - name: capacity 
        enabledAllocatable: true
        enablePreemptive: true
        enableReclaimable: true
        enableQueueOrder: true
        enableHierarchy: true
      - name: nodeorder
      - name: binpack
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: volcano
    meta.helm.sh/release-namespace: volcano-system
  creationTimestamp: "2025-07-22T07:47:05Z"
  labels:
    app.kubernetes.io/managed-by: Helm
  name: volcano-scheduler-configmap
  namespace: volcano-system
  resourceVersion: "51322036"
  uid: b77c408c-5873-4858-bcca-955b093bf750

2. queue
   create 3 queue: queue-my-root、child-queue-a、child-queue-b

kubectl get queue queue-my-root -o yaml

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"scheduling.volcano.sh/v1beta1","kind":"Queue","metadata":{"annotations":{},"name":"queue-my-root"},"spec":{"capability":{"cpu":"30","memory":"30Gi"},"deserved":{"cpu":30,"memory":"30Gi"},"parent":"r
oot","reclaimable":true}}
  creationTimestamp: "2025-09-25T10:14:21Z"
  generation: 1
  name: queue-my-root
  resourceVersion: "50912420"
  uid: 8e0e5d8f-0dd9-4fab-8b9b-527c08c21802
spec:
  capability:
    cpu: "30"
    memory: 30Gi
  deserved:
    cpu: 30
    memory: 30Gi
  parent: root
  reclaimable: true
  weight: 1
status:
  allocated:
    cpu: "0"
    memory: "0"
  reservation: {}
  state: Open




kubectl get queue child-queue-a -o yaml

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"scheduling.volcano.sh/v1beta1","kind":"Queue","metadata":{"annotations":{},"name":"child-queue-a"},"spec":{"capability":{"cpu":"20","memory":"20Gi"},"deserved":{"cpu":12,"memory":"12Gi"},"parent":"q
ueue-my-root","reclaimable":true}}
  creationTimestamp: "2025-09-25T10:14:21Z"
  generation: 1
  name: child-queue-a
  resourceVersion: "50912308"
  uid: 7f2bcfd4-2786-4382-86c4-885125500a29
spec:
  capability:
    cpu: "20"
    memory: 20Gi
  deserved:
    cpu: 12
    memory: 12Gi
  parent: queue-my-root
  reclaimable: true
  weight: 1
status:
  allocated:
    cpu: "0"
    memory: "0"
  reservation: {}
  state: Open




kubectl get queue child-queue-b -o yaml

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"scheduling.volcano.sh/v1beta1","kind":"Queue","metadata":{"annotations":{},"name":"child-queue-b"},"spec":{"capability":{"cpu":"30","memory":"30Gi"},"deserved":{"cpu":18,"memory":"18Gi"},"parent":"q
ueue-my-root","reclaimable":true}}
  creationTimestamp: "2025-09-25T10:14:21Z"
  generation: 1
  name: child-queue-b
  resourceVersion: "50912418"
  uid: ac8d9f53-dbc8-4dbd-90ad-ff745465f705
spec:
  capability:
    cpu: "30"
    memory: 30Gi
  deserved:
    cpu: 18
    memory: 18Gi
  parent: queue-my-root
  reclaimable: true
  weight: 1
status:
  allocated:
    cpu: "0"
    memory: "0"
  reservation: {}
  state: Open

3. SparkApplication

3.1 submit first SparkApplication to queue child-queue-a

kubectl apply -f app1.yaml

cat app1.yaml

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pi-1-demo
  namespace: my-spark-operator-ha-apps-ns
spec:
  type: Java
  mode: cluster
  image: harbor.demo-registry.cn/library/apache/spark:v3.4.1_arm_4
  imagePullPolicy: IfNotPresent
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.4.1.jar
  batchScheduler: volcano
  batchSchedulerOptions:
    queue: child-queue-a
    resources:
      cpu: "16"
      memory: "16Gi"
  sparkConf:
    spark.jars.ivy: "/tmp/.ivy"
    spark.app.name: "spark-pi-1-demo"
    spark.kubernetes.executor.podNamePrefix: "spark-pi-1-demo"
  arguments:
    - "500000"
  sparkVersion: 3.4.1
  driver:
    labels:
      version: 3.4.1
    cores: 2
    memory: 1664m
    serviceAccount: my-spark-operator-ha-release1-spark
  executor:
    labels:
      version: 3.4.1
    instances: 7
    cores: 2
    memory: 1664m

3.2 submit second SparkApplication to queue child-queue-b

kubectl apply -f app3.yaml

cat app3.yaml

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pi-3-demo
  namespace: my-spark-operator-ha-apps-ns
spec:
  type: Java
  mode: cluster
  image: harbor.demo-registry.cn/library/apache/spark:v3.4.1_arm_4
  imagePullPolicy: IfNotPresent
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.4.1.jar
  batchScheduler: volcano
  batchSchedulerOptions:
    queue: child-queue-b
    resources:
      cpu: "16"
      memory: "16Gi"
  sparkConf:
    spark.jars.ivy: "/tmp/.ivy"
    spark.app.name: "spark-pi-3-demo"
    spark.kubernetes.executor.podNamePrefix: "spark-pi-3-demo"
  arguments:
    - "500000"
  sparkVersion: 3.4.1
  driver:
    labels:
      version: 3.4.1
    cores: 2
    memory: 1664m
    serviceAccount: my-spark-operator-ha-release1-spark
  executor:
    labels:
      version: 3.4.1
    instances: 7
    cores: 2
    memory: 1664m

Describe the results you received and expected

1. Received result

1.1 after submit first SparkApplication
the first SparkApplication submit to the queue child-queue-a successfully

kubectl get all -n my-spark-operator-ha-apps-ns -o wide

NAME                         READY   STATUS    RESTARTS   AGE   IP             NODE                          NOMINATED NODE   READINESS GATES
pod/spark-pi-1-demo-driver   1/1     Running   0          20s   10.xxx.xxx.24    218   <none>           <none>
pod/spark-pi-1-demo-exec-1   1/1     Running   0          12s   10.xxx.xxx.178   225   <none>           <none>
pod/spark-pi-1-demo-exec-2   1/1     Running   0          12s   10.xxx.xxx.145   219   <none>           <none>
pod/spark-pi-1-demo-exec-3   1/1     Running   0          11s   10.xxx.xxx.25    218   <none>           <none>
pod/spark-pi-1-demo-exec-4   1/1     Running   0          11s   10.xxx.xxx.143   224   <none>           <none>
pod/spark-pi-1-demo-exec-5   1/1     Running   0          11s   10.xxx.xxx.179   225   <none>           <none>
pod/spark-pi-1-demo-exec-6   1/1     Running   0          10s   10.xxx.xxx.146   219   <none>           <none>
pod/spark-pi-1-demo-exec-7   1/1     Running   0          10s   10.xxx.xxx.26    218   <none>           <none>


kubectl get queue child-queue-a -o yaml

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"scheduling.volcano.sh/v1beta1","kind":"Queue","metadata":{"annotations":{},"name":"child-queue-a"},"spec":{"capability":{"cpu":"20","memory":"20Gi"},"deserved":{"cpu":12,"memory":"12Gi"},"parent":"q
ueue-my-root","reclaimable":true}}
  creationTimestamp: "2025-09-25T10:14:21Z"
  generation: 1
  name: child-queue-a
  resourceVersion: "51338783"
  uid: 7f2bcfd4-2786-4382-86c4-885125500a29
spec:
  capability:
    cpu: "20"
    memory: 20Gi
  deserved:
    cpu: 12
    memory: 12Gi
  parent: queue-my-root
  reclaimable: true
  weight: 1
status:
  allocated:
    cpu: "16"
    memory: 16Gi
    pods: "8"
  reservation: {}
  state: Open


kubectl get queue child-queue-b -o yaml

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"scheduling.volcano.sh/v1beta1","kind":"Queue","metadata":{"annotations":{},"name":"child-queue-b"},"spec":{"capability":{"cpu":"30","memory":"30Gi"},"deserved":{"cpu":18,"memory":"18Gi"},"parent":"q
ueue-my-root","reclaimable":true}}
  creationTimestamp: "2025-09-25T10:14:21Z"
  generation: 1
  name: child-queue-b
  resourceVersion: "51337693"
  uid: ac8d9f53-dbc8-4dbd-90ad-ff745465f705
spec:
  capability:
    cpu: "30"
    memory: 30Gi
  deserved:
    cpu: 18
    memory: 18Gi
  parent: queue-my-root
  reclaimable: true
  weight: 1
status:
  allocated:
    cpu: "0"
    memory: "0"
  reservation: {}
  state: Open


kubectl get podgroup -n my-spark-operator-ha-apps-ns spark-spark-pi-1-demo-pg -o yaml 

apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
  creationTimestamp: "2025-10-10T10:06:58Z"
  generation: 8
  name: spark-spark-pi-1-demo-pg
  namespace: my-spark-operator-ha-apps-ns
  ownerReferences:
  - apiVersion: sparkoperator.k8s.io/v1beta2
    blockOwnerDeletion: true
    controller: true
    kind: SparkApplication
    name: spark-pi-1-demo
    uid: 44f44722-00c1-47e9-b0a8-9bd6445be656
  resourceVersion: "51338871"
  uid: 3d4c61e2-c11b-4f17-ba58-82beaf095ce2
spec:
  minMember: 1
  minResources:
    cpu: "16"
    memory: 16Gi
  queue: child-queue-a
status:
  conditions:
  - lastTransitionTime: "2025-10-10T10:07:20Z"
    reason: tasks in gang are ready to be scheduled
    status: "True"
    transitionID: d6e7b2a1-c330-4426-84be-dc279869dca7
    type: Scheduled
  phase: Running
  running: 8

1.2 after submit second SparkApplication
however, the second SparkApplication is pending, and cannot reclaim resource from the queue child-queue-a !!!

kubectl get all -n my-spark-operator-ha-apps-ns -o wide

NAME                         READY   STATUS    RESTARTS   AGE   IP             NODE                          NOMINATED NODE   READINESS GATES
pod/spark-pi-1-demo-driver   1/1     Running   0          73s   10.xxx.xxx.24    218   <none>           <none>
pod/spark-pi-1-demo-exec-1   1/1     Running   0          65s   10.xxx.xxx.178   225   <none>           <none>
pod/spark-pi-1-demo-exec-2   1/1     Running   0          65s   10.xxx.xxx.145   219   <none>           <none>
pod/spark-pi-1-demo-exec-3   1/1     Running   0          64s   10.xxx.xxx.25    218   <none>           <none>
pod/spark-pi-1-demo-exec-4   1/1     Running   0          64s   10.xxx.xxx.143   224   <none>           <none>
pod/spark-pi-1-demo-exec-5   1/1     Running   0          64s   10.xxx.xxx.179   225   <none>           <none>
pod/spark-pi-1-demo-exec-6   1/1     Running   0          63s   10.xxx.xxx.146   219   <none>           <none>
pod/spark-pi-1-demo-exec-7   1/1     Running   0          63s   10.xxx.xxx.26    218   <none>           <none>
pod/spark-pi-3-demo-driver   0/1     Pending   0          3s    <none>         <none>                        <none>           <none>



kubectl get queue child-queue-a -o yaml

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"scheduling.volcano.sh/v1beta1","kind":"Queue","metadata":{"annotations":{},"name":"child-queue-a"},"spec":{"capability":{"cpu":"20","memory":"20Gi"},"deserved":{"cpu":12,"memory":"12Gi"},"parent":"q
ueue-my-root","reclaimable":true}}
  creationTimestamp: "2025-09-25T10:14:21Z"
  generation: 1
  name: child-queue-a
  resourceVersion: "51338783"
  uid: 7f2bcfd4-2786-4382-86c4-885125500a29
spec:
  capability:
    cpu: "20"
    memory: 20Gi
  deserved:
    cpu: 12
    memory: 12Gi
  parent: queue-my-root
  reclaimable: true
  weight: 1
status:
  allocated:
    cpu: "16"
    memory: 16Gi
    pods: "8"
  reservation: {}
  state: Open




kubectl get queue child-queue-b -o yaml

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"scheduling.volcano.sh/v1beta1","kind":"Queue","metadata":{"annotations":{},"name":"child-queue-b"},"spec":{"capability":{"cpu":"30","memory":"30Gi"},"deserved":{"cpu":18,"memory":"18Gi"},"parent":"q
ueue-my-root","reclaimable":true}}
  creationTimestamp: "2025-09-25T10:14:21Z"
  generation: 1
  name: child-queue-b
  resourceVersion: "51337693"
  uid: ac8d9f53-dbc8-4dbd-90ad-ff745465f705
spec:
  capability:
    cpu: "30"
    memory: 30Gi
  deserved:
    cpu: 18
    memory: 18Gi
  parent: queue-my-root
  reclaimable: true
  weight: 1
status:
  allocated:
    cpu: "0"
    memory: "0"
  reservation: {}
  state: Open

kubectl get podgroup -n my-spark-operator-ha-apps-ns spark-spark-pi-1-demo-pg -o yaml

apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
  creationTimestamp: "2025-10-10T10:06:58Z"
  generation: 8
  name: spark-spark-pi-1-demo-pg
  namespace: my-spark-operator-ha-apps-ns
  ownerReferences:
  - apiVersion: sparkoperator.k8s.io/v1beta2
    blockOwnerDeletion: true
    controller: true
    kind: SparkApplication
    name: spark-pi-1-demo
    uid: 44f44722-00c1-47e9-b0a8-9bd6445be656
  resourceVersion: "51338871"
  uid: 3d4c61e2-c11b-4f17-ba58-82beaf095ce2
spec:
  minMember: 1
  minResources:
    cpu: "16"
    memory: 16Gi
  queue: child-queue-a
status:
  conditions:
  - lastTransitionTime: "2025-10-10T10:07:20Z"
    reason: tasks in gang are ready to be scheduled
    status: "True"
    transitionID: d6e7b2a1-c330-4426-84be-dc279869dca7
    type: Scheduled
  phase: Running
  running: 8



kubectl get podgroup -n my-spark-operator-ha-apps-ns spark-spark-pi-3-demo-pg -o yaml

apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
  creationTimestamp: "2025-10-10T10:08:07Z"
  generation: 2
  name: spark-spark-pi-3-demo-pg
  namespace: my-spark-operator-ha-apps-ns
  ownerReferences:
  - apiVersion: sparkoperator.k8s.io/v1beta2
    blockOwnerDeletion: true
    controller: true
    kind: SparkApplication
    name: spark-pi-3-demo
    uid: 4b2bc8ef-42f6-4427-b8fb-6699e777ac98
  resourceVersion: "51339167"
  uid: 58058948-b87f-4ed5-bcb2-dc68b66a3b59
spec:
  minMember: 1
  minResources:
    cpu: "16"
    memory: 16Gi
  queue: child-queue-b
status:
  conditions:
  - lastTransitionTime: "2025-10-10T10:08:13Z"
    message: '1/1 tasks in gang unschedulable: pod group is not ready, 1 Pending,
      1 minAvailable; Pending: 1 Unschedulable'
    reason: NotEnoughResources
    status: "True"
    transitionID: 41d4926a-8f8f-4321-b405-0c2b799560b1
    type: Unschedulable
  phase: Pending

2. Expected result

after submit second SparkApplication,
the second SparkApplication can reclaim resource from the queue child-queue-a.
in the queue child-queue-a, there will be 1 driver running, 6 executor running, 1 executor pending.
in the queue child-queue-b, there will be 1 driver running, 7 executor running.

What version of Volcano are you using?

1.12.1

Any other relevant information

Environment :
k8s 1.24.10
spark operator 2.2.0
spark 3.4.1
volcano 1.12.1

arch : arm

[hajnalmt]

Hello @Jylx,
Can you send us some logs about the scheduler on the verbosity level of 5?

You can set it in the deployment directly by modifying the args part in the container to 5 from 3.
- -v=5

Or temporarily with curl inside the scheduler pod:

sudo curl --unix-socket /tmp/socks/klog.sock "http://localhost/setlevel?level=5&duration=60s"

[hajnalmt]

One improvement until you send the log can be to at least switch the action logic in the configuration so preemption action happens after reclaim.
I mean changing the configuration to:

apiVersion: v1
data:
  volcano-scheduler.conf: |
    actions: "enqueue, allocate, reclaim, preempt, backfill" 

reclaim, preempt

[Johnny539]

set - -v=5, watch log "Victims from", for more information

or maybe add following annotation on podgroup/pod

volcano.sh/preemptable=tue

[Jylx]

Hello @hajnalmt ,
Thank you for your suggestions very much.

1、I made the following two modifications

(1) update the volcano-scheduler configmaps
kubectl edit configmaps -n volcano-system volcano-scheduler-configmap

actions: "enqueue, allocate, reclaim, preempt, backfill"

kubectl -n volcano-system rollout restart deployment volcano-scheduler

(2) update the verbosity level to 5
kubectl edit deployments -n volcano-system volcano-scheduler

kubectl -n volcano-system rollout restart deployment volcano-scheduler

kubectl get deployments -n volcano-system volcano-scheduler -o yaml

  spec:
      containers:
      - args:
        - --logtostderr
        - --scheduler-conf=/volcano.scheduler/volcano-scheduler.conf
        - --enable-healthz=true
        - --enable-metrics=true
        - --leader-elect=false
        - --kube-api-qps=2000
        - --kube-api-burst=2000
        - --schedule-period=1s
        - --node-worker-threads=20
        - --plugins-dir=/plugins
        - -v=5
        - 2>&1

2、and then , the log are

(1)、after submit first SparkApplication

I1010 13:49:21.221874       1 scheduler.go:101] Start scheduling ...
I1010 13:49:21.221998       1 node_info.go:229] imageStates is map[]
I1010 13:49:21.222079       1 node_info.go:229] imageStates is map[]
I1010 13:49:21.222155       1 node_info.go:229] imageStates is map[]
I1010 13:49:21.222260       1 node_info.go:229] imageStates is map[]
I1010 13:49:21.222339       1 node_info.go:229] imageStates is map[]
I1010 13:49:21.222445       1 node_info.go:229] imageStates is map[]
I1010 13:49:21.222510       1 cache.go:1391] The priority of job <my-spark-operator-ha-apps-ns/spark-spark-pi-1-demo-pg> is </0>
I1010 13:49:21.222590       1 cache.go:1427] "SnapShot for scheduling" jobNum=1 QueueNum=5 NodeNum=6
I1010 13:49:21.222619       1 cache.go:1430] "HyperNode snapShot for scheduling" tiers={} realNodesSet={} hyperNodesReadyToSchedule=false
I1010 13:49:21.222633       1 session.go:228] Add node 222 resources: cpu 16000.00, memory 32780976128.00, pods 110.00, ephemeral-storage 85124969946000.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00
I1010 13:49:21.222663       1 session.go:228] Add node 225 resources: cpu 16000.00, memory 32780976128.00, hugepages-512Mi 0.00, pods 110.00, ephemeral-storage 85124969946000.00, hugepages-2Mi 0.00
I1010 13:49:21.222674       1 session.go:228] Add node 219 resources: cpu 16000.00, memory 32780976128.00, pods 110.00, ephemeral-storage 85124969946000.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00
I1010 13:49:21.222686       1 session.go:228] Add node 224 resources: cpu 16000.00, memory 32780976128.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00, pods 110.00, ephemeral-storage 85124969946000.00
I1010 13:49:21.222696       1 session.go:228] Add node 218 resources: cpu 16000.00, memory 32780976128.00, pods 110.00, ephemeral-storage 85124969946000.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00
I1010 13:49:21.222712       1 session.go:228] Add node 223 resources: cpu 16000.00, memory 32780976128.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00, pods 110.00, ephemeral-storage 85124969946000.00
I1010 13:49:21.222729       1 session.go:234] Open Session 1ba52d50-5a5e-4d80-85c2-37946877ba74 with <1> Job and <5> Queues
I1010 13:49:21.223388       1 overcommit.go:75] Enter overcommit plugin ...
I1010 13:49:21.223422       1 overcommit.go:142] Leaving overcommit plugin.
I1010 13:49:21.223434       1 drf.go:185] Total Allocatable cpu 96000.00, memory 196685856768.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00, pods 660.00, ephemeral-storage 510749819676000.00
I1010 13:49:21.223517       1 factory.go:43] Register preBinder predicates successfully
I1010 13:49:21.223534       1 capacity.go:101] The total resource is <cpu 96000.00, memory 196685856768.00, ephemeral-storage 510749819676000.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00, pods 660.00>
I1010 13:49:21.223562       1 capacity.go:450] Considering Job <my-spark-operator-ha-apps-ns/spark-spark-pi-1-demo-pg>.
I1010 13:49:21.223601       1 capacity.go:498] Queue child-queue-a allocated <cpu 16000.00, memory 17179869184.00, pods 8.00> request <cpu 16000.00, memory 17179869184.00, pods 8.00> inqueue <cpu 0.00, memory 0.00> elastic <cpu 0.00, memory 0.00, pods 8.00>
I1010 13:49:21.223630       1 capacity.go:513] Successfully checked queue's hierarchical structure.
I1010 13:49:21.223642       1 capacity.go:521] The attributes of queue <default> in capacity: deserved <cpu 0.00, memory 0.00>, realCapability <cpu 96000.00, memory 196685856768.00, ephemeral-storage 510749819676000.00, pods 660.00>, allocate <cpu 0.00, memory 0.00>, request <cpu 0.00, memory 0.00>, elastic <cpu 0.00, memory 0.00>, share <0.00>
I1010 13:49:21.223661       1 capacity.go:521] The attributes of queue <child-queue-a> in capacity: deserved <cpu 12000.00, memory 12884901888.00>, realCapability <cpu 20000.00, memory 21474836480.00, pods 660.00, ephemeral-storage 510749819676000.00>, allocate <cpu 16000.00, memory 17179869184.00, pods 8.00>, request <cpu 16000.00, memory 17179869184.00, pods 8.00>, elastic <cpu 0.00, memory 0.00, pods 8.00>, share <1.33>
I1010 13:49:21.223684       1 capacity.go:521] The attributes of queue <child-queue-b> in capacity: deserved <cpu 18000.00, memory 19327352832.00>, realCapability <cpu 30000.00, memory 32212254720.00, ephemeral-storage 510749819676000.00, pods 660.00>, allocate <cpu 0.00, memory 0.00>, request <cpu 0.00, memory 0.00>, elastic <cpu 0.00, memory 0.00>, share <0.00>
I1010 13:49:21.223702       1 capacity.go:521] The attributes of queue <queue-my-root> in capacity: deserved <cpu 30000.00, memory 32212254720.00>, realCapability <cpu 30000.00, memory 32212254720.00, pods 660.00, ephemeral-storage 510749819676000.00>, allocate <cpu 16000.00, memory 17179869184.00, pods 8.00>, request <cpu 16000.00, memory 17179869184.00, pods 8.00>, elastic <cpu 0.00, memory 0.00, pods 8.00>, share <0.53>
I1010 13:49:21.223723       1 capacity.go:521] The attributes of queue <root> in capacity: deserved <cpu 96000.00, memory 196685856768.00, pods 660.00, ephemeral-storage 510749819676000.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00>, realCapability <cpu 96000.00, memory 196685856768.00, hugepages-512Mi 0.00, pods 660.00, ephemeral-storage 510749819676000.00, hugepages-2Mi 0.00>, allocate <cpu 16000.00, memory 17179869184.00, pods 8.00>, request <cpu 16000.00, memory 17179869184.00, pods 8.00>, elastic <cpu 0.00, memory 0.00, pods 8.00>, share <0.17>
I1010 13:49:21.223796       1 binpack.go:165] Enter binpack plugin ...
I1010 13:49:21.223810       1 binpack.go:183] resources [] record in weight but not found on any node
I1010 13:49:21.223827       1 binpack.go:167] Leaving binpack plugin. binpack.weight[1], binpack.cpu[1], binpack.memory[1], memory[1], cpu[1] ...
I1010 13:49:21.223838       1 enqueue.go:45] Enter Enqueue ...
I1010 13:49:21.223846       1 enqueue.go:63] Added Queue <child-queue-a> for Job <my-spark-operator-ha-apps-ns/spark-spark-pi-1-demo-pg>
I1010 13:49:21.223858       1 enqueue.go:79] Try to enqueue PodGroup to 0 Queues
I1010 13:49:21.223871       1 enqueue.go:104] Leaving Enqueue ...
I1010 13:49:21.223884       1 allocate.go:77] Enter Allocate ...
I1010 13:49:21.223896       1 job_info.go:929] job spark-spark-pi-1-demo-pg/my-spark-operator-ha-apps-ns actual: map[:8], ji.TaskMinAvailable: map[]
I1010 13:49:21.223911       1 allocate.go:132] Added Job <my-spark-operator-ha-apps-ns/spark-spark-pi-1-demo-pg> into Queue <child-queue-a>
I1010 13:49:21.223917       1 allocate.go:96] Try to allocate resource to 1 Queues
I1010 13:49:21.223924       1 allocate.go:161] Try to allocate resource to Jobs in Queue <child-queue-a>
I1010 13:49:21.223931       1 allocate.go:161] Try to allocate resource to Jobs in Queue <child-queue-a>
I1010 13:49:21.223937       1 allocate.go:165] Can not find jobs for queue child-queue-a.
I1010 13:49:21.223942       1 allocate.go:98] Leaving Allocate ...
I1010 13:49:21.223950       1 reclaim.go:41] Enter Reclaim ...
I1010 13:49:21.223961       1 reclaim.go:50] There are <1> Jobs and <5> Queues in total for scheduling.
I1010 13:49:21.223967       1 job_info.go:929] job spark-spark-pi-1-demo-pg/my-spark-operator-ha-apps-ns actual: map[:8], ji.TaskMinAvailable: map[]
I1010 13:49:21.223980       1 reclaim.go:67] Added Queue <child-queue-a> for Job <my-spark-operator-ha-apps-ns/spark-spark-pi-1-demo-pg>
I1010 13:49:21.223991       1 reclaim.go:228] Leaving Reclaim ...
I1010 13:49:21.224001       1 preempt.go:54] Enter Preempt ...
I1010 13:49:21.224007       1 job_info.go:929] job spark-spark-pi-1-demo-pg/my-spark-operator-ha-apps-ns actual: map[:8], ji.TaskMinAvailable: map[]
I1010 13:49:21.224020       1 preempt.go:78] Added Queue <child-queue-a> for Job <my-spark-operator-ha-apps-ns/spark-spark-pi-1-demo-pg>
I1010 13:49:21.224028       1 preempt.go:108] No preemptors in Queue <child-queue-a>, break.
I1010 13:49:21.224034       1 preempt.go:221] Leaving Preempt ...
I1010 13:49:21.224045       1 backfill.go:53] Enter Backfill ...
I1010 13:49:21.224053       1 job_info.go:929] job spark-spark-pi-1-demo-pg/my-spark-operator-ha-apps-ns actual: map[:8], ji.TaskMinAvailable: map[]
I1010 13:49:21.224066       1 backfill.go:104] Leaving Backfill ...
I1010 13:49:21.224181       1 session.go:297] Queue <child-queue-b> allocated resource keeps equal, no need to update queue status <map[cpu:{{0 0} {<nil>} 0 DecimalSI} memory:{{0 0} {<nil>} 0 DecimalSI}]>.
I1010 13:49:21.224226       1 session.go:319] Root queue deserved/guaranteed resource and allocated resource remains the same, no need to update the queue.
I1010 13:49:21.224243       1 session.go:297] Queue <default> allocated resource keeps equal, no need to update queue status <map[cpu:{{0 0} {<nil>} 0 DecimalSI} memory:{{0 0} {<nil>} 0 DecimalSI}]>.
I1010 13:49:21.224281       1 session.go:297] Queue <queue-my-root> allocated resource keeps equal, no need to update queue status <map[cpu:{{16 0} {<nil>} 16 DecimalSI} memory:{{17179869184 0} {<nil>}  BinarySI} pods:{{8 0} {<nil>} 8 DecimalSI}]>.
I1010 13:49:21.224305       1 session.go:297] Queue <child-queue-a> allocated resource keeps equal, no need to update queue status <map[cpu:{{16 0} {<nil>} 16 DecimalSI} memory:{{17179869184 0} {<nil>}  BinarySI} pods:{{8 0} {<nil>} 8 DecimalSI}]>.
I1010 13:49:21.224315       1 session.go:368] Close Session 1ba52d50-5a5e-4d80-85c2-37946877ba74
I1010 13:49:21.224332       1 scheduler.go:128] End scheduling ...

(2)、after submit second SparkApplication

I1010 13:50:05.413331       1 scheduler.go:101] Start scheduling ...
I1010 13:50:05.413554       1 node_info.go:229] imageStates is map[]
I1010 13:50:05.413651       1 node_info.go:229] imageStates is map[]
I1010 13:50:05.413743       1 node_info.go:229] imageStates is map[]
I1010 13:50:05.413808       1 node_info.go:229] imageStates is map[]
I1010 13:50:05.413866       1 node_info.go:229] imageStates is map[]
I1010 13:50:05.413935       1 node_info.go:229] imageStates is map[]
I1010 13:50:05.413995       1 cache.go:1391] The priority of job <my-spark-operator-ha-apps-ns/spark-spark-pi-3-demo-pg> is </0>
I1010 13:50:05.414048       1 cache.go:1391] The priority of job <my-spark-operator-ha-apps-ns/spark-spark-pi-1-demo-pg> is </0>
I1010 13:50:05.414188       1 cache.go:1427] "SnapShot for scheduling" jobNum=2 QueueNum=5 NodeNum=6
I1010 13:50:05.414220       1 cache.go:1430] "HyperNode snapShot for scheduling" tiers={} realNodesSet={} hyperNodesReadyToSchedule=false
I1010 13:50:05.414242       1 session.go:228] Add node 219 resources: cpu 16000.00, memory 32780976128.00, ephemeral-storage 85124969946000.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00, pods 110.00
I1010 13:50:05.414267       1 session.go:228] Add node 224 resources: cpu 16000.00, memory 32780976128.00, ephemeral-storage 85124969946000.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00, pods 110.00
I1010 13:50:05.414288       1 session.go:228] Add node 218 resources: cpu 16000.00, memory 32780976128.00, pods 110.00, ephemeral-storage 85124969946000.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00
I1010 13:50:05.414305       1 session.go:228] Add node 223 resources: cpu 16000.00, memory 32780976128.00, pods 110.00, ephemeral-storage 85124969946000.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00
I1010 13:50:05.414321       1 session.go:228] Add node 222 resources: cpu 16000.00, memory 32780976128.00, pods 110.00, ephemeral-storage 85124969946000.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00
I1010 13:50:05.414332       1 session.go:228] Add node 225 resources: cpu 16000.00, memory 32780976128.00, pods 110.00, ephemeral-storage 85124969946000.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00
I1010 13:50:05.414350       1 session.go:234] Open Session 0ec2f322-58dc-4e34-b123-a634c0c7f0fa with <2> Job and <5> Queues
I1010 13:50:05.415090       1 overcommit.go:75] Enter overcommit plugin ...
I1010 13:50:05.415123       1 overcommit.go:142] Leaving overcommit plugin.
I1010 13:50:05.415137       1 drf.go:185] Total Allocatable cpu 96000.00, memory 196685856768.00, ephemeral-storage 510749819676000.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00, pods 660.00
I1010 13:50:05.415222       1 factory.go:43] Register preBinder predicates successfully
I1010 13:50:05.415239       1 capacity.go:101] The total resource is <cpu 96000.00, memory 196685856768.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00, pods 660.00, ephemeral-storage 510749819676000.00>
I1010 13:50:05.415268       1 capacity.go:450] Considering Job <my-spark-operator-ha-apps-ns/spark-spark-pi-3-demo-pg>.
I1010 13:50:05.415298       1 capacity.go:498] Queue child-queue-b allocated <cpu 0.00, memory 0.00> request <cpu 2000.00, memory 2147483648.00, pods 1.00> inqueue <cpu 0.00, memory 0.00> elastic <cpu 0.00, memory 0.00>
I1010 13:50:05.415309       1 capacity.go:450] Considering Job <my-spark-operator-ha-apps-ns/spark-spark-pi-1-demo-pg>.
I1010 13:50:05.415338       1 capacity.go:498] Queue child-queue-a allocated <cpu 16000.00, memory 17179869184.00, pods 8.00> request <cpu 16000.00, memory 17179869184.00, pods 8.00> inqueue <cpu 0.00, memory 0.00> elastic <cpu 0.00, memory 0.00, pods 8.00>
I1010 13:50:05.415368       1 capacity.go:513] Successfully checked queue's hierarchical structure.
I1010 13:50:05.415380       1 capacity.go:521] The attributes of queue <queue-my-root> in capacity: deserved <cpu 30000.00, memory 32212254720.00>, realCapability <cpu 30000.00, memory 32212254720.00, pods 660.00, ephemeral-storage 510749819676000.00>, allocate <cpu 16000.00, memory 17179869184.00, pods 8.00>, request <cpu 18000.00, memory 19327352832.00, pods 9.00>, elastic <cpu 0.00, memory 0.00, pods 8.00>, share <0.53>
I1010 13:50:05.415405       1 capacity.go:521] The attributes of queue <child-queue-a> in capacity: deserved <cpu 12000.00, memory 12884901888.00>, realCapability <cpu 20000.00, memory 21474836480.00, pods 660.00, ephemeral-storage 510749819676000.00>, allocate <cpu 16000.00, memory 17179869184.00, pods 8.00>, request <cpu 16000.00, memory 17179869184.00, pods 8.00>, elastic <cpu 0.00, memory 0.00, pods 8.00>, share <1.33>
I1010 13:50:05.415427       1 capacity.go:521] The attributes of queue <child-queue-b> in capacity: deserved <cpu 18000.00, memory 19327352832.00>, realCapability <cpu 30000.00, memory 32212254720.00, pods 660.00, ephemeral-storage 510749819676000.00>, allocate <cpu 0.00, memory 0.00>, request <cpu 2000.00, memory 2147483648.00, pods 1.00>, elastic <cpu 0.00, memory 0.00>, share <0.00>
I1010 13:50:05.415446       1 capacity.go:521] The attributes of queue <root> in capacity: deserved <cpu 96000.00, memory 196685856768.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00, pods 660.00, ephemeral-storage 510749819676000.00>, realCapability <cpu 96000.00, memory 196685856768.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00, pods 660.00, ephemeral-storage 510749819676000.00>, allocate <cpu 16000.00, memory 17179869184.00, pods 8.00>, request <cpu 18000.00, memory 19327352832.00, pods 9.00>, elastic <cpu 0.00, memory 0.00, pods 8.00>, share <0.17>
I1010 13:50:05.415472       1 capacity.go:521] The attributes of queue <default> in capacity: deserved <cpu 0.00, memory 0.00>, realCapability <cpu 96000.00, memory 196685856768.00, pods 660.00, ephemeral-storage 510749819676000.00>, allocate <cpu 0.00, memory 0.00>, request <cpu 0.00, memory 0.00>, elastic <cpu 0.00, memory 0.00>, share <0.00>
I1010 13:50:05.415540       1 binpack.go:165] Enter binpack plugin ...
I1010 13:50:05.415554       1 binpack.go:183] resources [] record in weight but not found on any node
I1010 13:50:05.415564       1 binpack.go:167] Leaving binpack plugin. binpack.weight[1], binpack.cpu[1], binpack.memory[1], cpu[1], memory[1] ...
I1010 13:50:05.415573       1 enqueue.go:45] Enter Enqueue ...
I1010 13:50:05.415585       1 enqueue.go:63] Added Queue <child-queue-b> for Job <my-spark-operator-ha-apps-ns/spark-spark-pi-3-demo-pg>
I1010 13:50:05.415617       1 enqueue.go:74] Added Job <my-spark-operator-ha-apps-ns/spark-spark-pi-3-demo-pg> into Queue <child-queue-b>
I1010 13:50:05.415630       1 enqueue.go:63] Added Queue <child-queue-a> for Job <my-spark-operator-ha-apps-ns/spark-spark-pi-1-demo-pg>
I1010 13:50:05.415639       1 enqueue.go:79] Try to enqueue PodGroup to 1 Queues
I1010 13:50:05.415649       1 overcommit.go:125] Sufficient resources, permit job <my-spark-operator-ha-apps-ns/spark-spark-pi-3-demo-pg> to be inqueue
I1010 13:50:05.415670       1 capacity.go:767] job spark-spark-pi-3-demo-pg min resource <cpu 16000.00, memory 17179869184.00>, queue child-queue-b capability <cpu 30000.00, memory 32212254720.00, pods 660.00, ephemeral-storage 510749819676000.00> allocated <cpu 0.00, memory 0.00> inqueue <cpu 0.00, memory 0.00> elastic <cpu 0.00, memory 0.00>
I1010 13:50:05.415686       1 capacity.go:767] job spark-spark-pi-3-demo-pg min resource <cpu 16000.00, memory 17179869184.00>, queue queue-my-root capability <cpu 30000.00, memory 32212254720.00, pods 660.00, ephemeral-storage 510749819676000.00> allocated <cpu 16000.00, memory 17179869184.00, pods 8.00> inqueue <cpu 0.00, memory 0.00> elastic <cpu 0.00, memory 0.00, pods 8.00>
I1010 13:50:05.415703       1 capacity.go:767] job spark-spark-pi-3-demo-pg min resource <cpu 16000.00, memory 17179869184.00>, queue root capability <cpu 96000.00, memory 196685856768.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00, pods 660.00, ephemeral-storage 510749819676000.00> allocated <cpu 16000.00, memory 17179869184.00, pods 8.00> inqueue <cpu 0.00, memory 0.00> elastic <cpu 0.00, memory 0.00, pods 8.00>
I1010 13:50:05.415747       1 enqueue.go:104] Leaving Enqueue ...
I1010 13:50:05.415761       1 allocate.go:77] Enter Allocate ...
I1010 13:50:05.415767       1 allocate.go:106] Job <my-spark-operator-ha-apps-ns/spark-spark-pi-3-demo-pg> Queue <child-queue-b> skip allocate, reason: job status is pending.
I1010 13:50:05.415778       1 job_info.go:929] job spark-spark-pi-1-demo-pg/my-spark-operator-ha-apps-ns actual: map[:8], ji.TaskMinAvailable: map[]
I1010 13:50:05.415799       1 allocate.go:132] Added Job <my-spark-operator-ha-apps-ns/spark-spark-pi-1-demo-pg> into Queue <child-queue-a>
I1010 13:50:05.415805       1 allocate.go:96] Try to allocate resource to 1 Queues
I1010 13:50:05.415811       1 allocate.go:161] Try to allocate resource to Jobs in Queue <child-queue-a>
I1010 13:50:05.415822       1 allocate.go:161] Try to allocate resource to Jobs in Queue <child-queue-a>
I1010 13:50:05.415828       1 allocate.go:165] Can not find jobs for queue child-queue-a.
I1010 13:50:05.415833       1 allocate.go:98] Leaving Allocate ...
I1010 13:50:05.415840       1 reclaim.go:41] Enter Reclaim ...
I1010 13:50:05.415851       1 reclaim.go:50] There are <2> Jobs and <5> Queues in total for scheduling.
I1010 13:50:05.415858       1 job_info.go:929] job spark-spark-pi-1-demo-pg/my-spark-operator-ha-apps-ns actual: map[:8], ji.TaskMinAvailable: map[]
I1010 13:50:05.415871       1 reclaim.go:67] Added Queue <child-queue-a> for Job <my-spark-operator-ha-apps-ns/spark-spark-pi-1-demo-pg>
I1010 13:50:05.415881       1 reclaim.go:228] Leaving Reclaim ...
I1010 13:50:05.415891       1 preempt.go:54] Enter Preempt ...
I1010 13:50:05.415897       1 job_info.go:929] job spark-spark-pi-1-demo-pg/my-spark-operator-ha-apps-ns actual: map[:8], ji.TaskMinAvailable: map[]
I1010 13:50:05.415906       1 preempt.go:78] Added Queue <child-queue-a> for Job <my-spark-operator-ha-apps-ns/spark-spark-pi-1-demo-pg>
I1010 13:50:05.415918       1 preempt.go:108] No preemptors in Queue <child-queue-a>, break.
I1010 13:50:05.415925       1 preempt.go:221] Leaving Preempt ...
I1010 13:50:05.415936       1 backfill.go:53] Enter Backfill ...
I1010 13:50:05.415942       1 job_info.go:929] job spark-spark-pi-1-demo-pg/my-spark-operator-ha-apps-ns actual: map[:8], ji.TaskMinAvailable: map[]
I1010 13:50:05.415957       1 backfill.go:104] Leaving Backfill ...
I1010 13:50:05.416182       1 cache.go:1010] task unscheduleable my-spark-operator-ha-apps-ns/spark-pi-3-demo-driver, message: 1/1 tasks in gang unschedulable: pod group is not ready, 1 Pending, 1 minAvailable; Pending: 1 Unschedulable, skip by no condition update
I1010 13:50:05.416297       1 session.go:319] Root queue deserved/guaranteed resource and allocated resource remains the same, no need to update the queue.
I1010 13:50:05.416342       1 session.go:297] Queue <default> allocated resource keeps equal, no need to update queue status <map[cpu:{{0 0} {<nil>} 0 DecimalSI} memory:{{0 0} {<nil>} 0 DecimalSI}]>.
I1010 13:50:05.416378       1 session.go:297] Queue <queue-my-root> allocated resource keeps equal, no need to update queue status <map[cpu:{{16 0} {<nil>} 16 DecimalSI} memory:{{17179869184 0} {<nil>}  BinarySI} pods:{{8 0} {<nil>} 8 DecimalSI}]>.
I1010 13:50:05.416407       1 session.go:297] Queue <child-queue-a> allocated resource keeps equal, no need to update queue status <map[cpu:{{16 0} {<nil>} 16 DecimalSI} memory:{{17179869184 0} {<nil>}  BinarySI} pods:{{8 0} {<nil>} 8 DecimalSI}]>.
I1010 13:50:05.416432       1 session.go:297] Queue <child-queue-b> allocated resource keeps equal, no need to update queue status <map[cpu:{{0 0} {<nil>} 0 DecimalSI} memory:{{0 0} {<nil>} 0 DecimalSI}]>.
I1010 13:50:05.416442       1 session.go:368] Close Session 0ec2f322-58dc-4e34-b123-a634c0c7f0fa
I1010 13:50:05.416462       1 scheduler.go:128] End scheduling ...

3、result is

kubectl get all -n my-spark-operator-ha-apps-ns -o wide 

NAME                         READY   STATUS    RESTARTS   AGE    
pod/spark-pi-1-demo-driver   1/1     Running   0          110s   
pod/spark-pi-1-demo-exec-1   1/1     Running   0          102s   
pod/spark-pi-1-demo-exec-2   1/1     Running   0          102s   
pod/spark-pi-1-demo-exec-3   1/1     Running   0          102s   
pod/spark-pi-1-demo-exec-4   1/1     Running   0          102s   
pod/spark-pi-1-demo-exec-5   1/1     Running   0          102s   
pod/spark-pi-1-demo-exec-6   1/1     Running   0          101s   
pod/spark-pi-1-demo-exec-7   1/1     Running   0          101s   
pod/spark-pi-3-demo-driver   0/1     Pending   0          77s    




kubectl get sparkapplications.sparkoperator.k8s.io -n my-spark-operator-ha-apps-ns

NAME              STATUS      ATTEMPTS   
spark-pi-1-demo   RUNNING     1          
spark-pi-3-demo   SUBMITTED   1        

[Jylx]

Hello, @Johnny539
Thank you for your suggestions very much.

1、I customize a podgroup, and add the annotation to the podgroup and driver executor pod, but the driver of the second SparkApplication is still pending.

2、cannot find victim from the volcano scheduler log

[Johnny539]

Hello, @Johnny539 Thank you for your suggestions very much.

1、I customize a podgroup, and add the annotation to the podgroup and driver executor pod, but the driver of the second SparkApplication is still pending.

2、cannot find victim from the volcano scheduler log

1、add annotation both pg spark-pi-1-demo&spark-pi-3-demo
and paste volcano log

[Jylx]

Hello, @Johnny539 Thank you for your suggestions very much.
1、I customize a podgroup, and add the annotation to the podgroup and driver executor pod, but the driver of the second SparkApplication is still pending.
2、cannot find victim from the volcano scheduler log

1、add annotation both pg spark-pi-1-demo&spark-pi-3-demo and paste volcano log

Thank you for your suggestions very much.
According to your request, I have obtained the following results.

1、script

cat app1.yaml

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pi-1-demo
  namespace: my-spark-operator-ha-apps-ns
spec:
  type: Java
  mode: cluster
  image: harbor.demo-registry.cn/library/apache/spark:v3.4.1_arm_4
  imagePullPolicy: IfNotPresent
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.4.1.jar
  batchScheduler: volcano
  sparkConf:
    spark.jars.ivy: "/tmp/.ivy"
    spark.app.name: "spark-pi-1-demo"
    spark.kubernetes.executor.podNamePrefix: "spark-pi-1-demo"
  arguments:
    - "500000"
  sparkVersion: 3.4.1
  driver:
    labels:
      version: 3.4.1
    annotations:
      scheduling.k8s.io/group-name: spark-pg-1
    cores: 2
    memory: 1664m
    serviceAccount: my-spark-operator-ha-release1-spark
  executor:
    labels:
      version: 3.4.1
    annotations:
      scheduling.k8s.io/group-name: spark-pg-1
    instances: 7
    cores: 2
    memory: 1664m

cat app3.yaml

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pi-3-demo
  namespace: my-spark-operator-ha-apps-ns
spec:
  type: Java
  mode: cluster
  image: harbor.demo-registry.cn/library/apache/spark:v3.4.1_arm_4
  imagePullPolicy: IfNotPresent
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.4.1.jar
  batchScheduler: volcano
  sparkConf:
    spark.jars.ivy: "/tmp/.ivy"
    spark.app.name: "spark-pi-3-demo"
    spark.kubernetes.executor.podNamePrefix: "spark-pi-3-demo"
  arguments:
    - "500000"
  sparkVersion: 3.4.1
  driver:
    labels:
      version: 3.4.1
    annotations:
      scheduling.k8s.io/group-name: spark-pg-2
    cores: 2
    memory: 1664m
    serviceAccount: my-spark-operator-ha-release1-spark
  executor:
    labels:
      version: 3.4.1
    annotations:
      scheduling.k8s.io/group-name: spark-pg-2
    instances: 7
    cores: 2
    memory: 1664m

cat spark-pg-1.yaml

apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
  name: spark-pg-1
  namespace: my-spark-operator-ha-apps-ns
  annotations:
    volcano.sh/preemptable: "true"
spec:
  queue: child-queue-a
  minMember: 1
  minResources:
    cpu: "16"
    memory: "16Gi"

cat spark-pg-2.yaml

apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
  name: spark-pg-2
  namespace: my-spark-operator-ha-apps-ns
  annotations:
    volcano.sh/preemptable: "true"
spec:
  #queue: subchild-queue-a1
  queue: child-queue-b
  minMember: 1
  minResources:
    cpu: "16"
    memory: "16Gi"

2、submit first application

kubectl apply -f spark-pg-1.yaml
kubectl apply -f app1.yaml

log is

I1011 04:13:29.659386       1 scheduler.go:101] Start scheduling ...
I1011 04:13:29.659624       1 node_info.go:229] imageStates is map[]
I1011 04:13:29.659782       1 node_info.go:229] imageStates is map[]
I1011 04:13:29.659974       1 node_info.go:229] imageStates is map[]
I1011 04:13:29.660107       1 node_info.go:229] imageStates is map[]
I1011 04:13:29.660156       1 node_info.go:229] imageStates is map[]
I1011 04:13:29.660202       1 node_info.go:229] imageStates is map[]
I1011 04:13:29.660277       1 cache.go:1391] The priority of job <my-spark-operator-ha-apps-ns/spark-pg-1> is </0>
I1011 04:13:29.660406       1 cache.go:1427] "SnapShot for scheduling" jobNum=1 QueueNum=5 NodeNum=6
I1011 04:13:29.660432       1 cache.go:1430] "HyperNode snapShot for scheduling" tiers={} realNodesSet={} hyperNodesReadyToSchedule=false
I1011 04:13:29.660448       1 session.go:228] Add node 222 resources: cpu 16000.00, memory 32780976128.00, ephemeral-storage 85124969946000.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00, pods 110.00
I1011 04:13:29.660468       1 session.go:228] Add node 219 resources: cpu 16000.00, memory 32780976128.00, pods 110.00, ephemeral-storage 85124969946000.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00
I1011 04:13:29.660480       1 session.go:228] Add node 218 resources: cpu 16000.00, memory 32780976128.00, hugepages-512Mi 0.00, pods 110.00, ephemeral-storage 85124969946000.00, hugepages-2Mi 0.00
I1011 04:13:29.660490       1 session.go:228] Add node 223 resources: cpu 16000.00, memory 32780976128.00, hugepages-512Mi 0.00, pods 110.00, ephemeral-storage 85124969946000.00, hugepages-2Mi 0.00
I1011 04:13:29.660501       1 session.go:228] Add node 224 resources: cpu 16000.00, memory 32780976128.00, pods 110.00, ephemeral-storage 85124969946000.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00
I1011 04:13:29.660510       1 session.go:228] Add node 225 resources: cpu 16000.00, memory 32780976128.00, hugepages-512Mi 0.00, pods 110.00, ephemeral-storage 85124969946000.00, hugepages-2Mi 0.00
I1011 04:13:29.660522       1 session.go:234] Open Session cc73d2e7-40bf-4f25-a9da-7977b7fbc3a0 with <1> Job and <5> Queues
I1011 04:13:29.661452       1 overcommit.go:75] Enter overcommit plugin ...
I1011 04:13:29.661476       1 overcommit.go:142] Leaving overcommit plugin.
I1011 04:13:29.661489       1 drf.go:185] Total Allocatable cpu 96000.00, memory 196685856768.00, pods 660.00, ephemeral-storage 510749819676000.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00
I1011 04:13:29.661578       1 factory.go:43] Register preBinder predicates successfully
I1011 04:13:29.661593       1 capacity.go:101] The total resource is <cpu 96000.00, memory 196685856768.00, ephemeral-storage 510749819676000.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00, pods 660.00>
I1011 04:13:29.661625       1 capacity.go:450] Considering Job <my-spark-operator-ha-apps-ns/spark-pg-1>.
I1011 04:13:29.661671       1 capacity.go:498] Queue child-queue-a allocated <cpu 16000.00, memory 17179869184.00, pods 8.00> request <cpu 16000.00, memory 17179869184.00, pods 8.00> inqueue <cpu 0.00, memory 0.00> elastic <cpu 0.00, memory 0.00, pods 8.00>
I1011 04:13:29.661706       1 capacity.go:513] Successfully checked queue's hierarchical structure.
I1011 04:13:29.661717       1 capacity.go:521] The attributes of queue <child-queue-a> in capacity: deserved <cpu 12000.00, memory 12884901888.00>, realCapability <cpu 20000.00, memory 21474836480.00, ephemeral-storage 510749819676000.00, pods 660.00>, allocate <cpu 16000.00, memory 17179869184.00, pods 8.00>, request <cpu 16000.00, memory 17179869184.00, pods 8.00>, elastic <cpu 0.00, memory 0.00, pods 8.00>, share <1.33>
I1011 04:13:29.661738       1 capacity.go:521] The attributes of queue <child-queue-b> in capacity: deserved <cpu 18000.00, memory 19327352832.00>, realCapability <cpu 30000.00, memory 32212254720.00, ephemeral-storage 510749819676000.00, pods 660.00>, allocate <cpu 0.00, memory 0.00>, request <cpu 0.00, memory 0.00>, elastic <cpu 0.00, memory 0.00>, share <0.00>
I1011 04:13:29.661754       1 capacity.go:521] The attributes of queue <root> in capacity: deserved <cpu 96000.00, memory 196685856768.00, ephemeral-storage 510749819676000.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00, pods 660.00>, realCapability <cpu 96000.00, memory 196685856768.00, ephemeral-storage 510749819676000.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00, pods 660.00>, allocate <cpu 16000.00, memory 17179869184.00, pods 8.00>, request <cpu 16000.00, memory 17179869184.00, pods 8.00>, elastic <cpu 0.00, memory 0.00, pods 8.00>, share <0.17>
I1011 04:13:29.661776       1 capacity.go:521] The attributes of queue <default> in capacity: deserved <cpu 0.00, memory 0.00>, realCapability <cpu 96000.00, memory 196685856768.00, pods 660.00, ephemeral-storage 510749819676000.00>, allocate <cpu 0.00, memory 0.00>, request <cpu 0.00, memory 0.00>, elastic <cpu 0.00, memory 0.00>, share <0.00>
I1011 04:13:29.661789       1 capacity.go:521] The attributes of queue <queue-my-root> in capacity: deserved <cpu 30000.00, memory 32212254720.00>, realCapability <cpu 30000.00, memory 32212254720.00, ephemeral-storage 510749819676000.00, pods 660.00>, allocate <cpu 16000.00, memory 17179869184.00, pods 8.00>, request <cpu 16000.00, memory 17179869184.00, pods 8.00>, elastic <cpu 0.00, memory 0.00, pods 8.00>, share <0.53>
I1011 04:13:29.661859       1 binpack.go:165] Enter binpack plugin ...
I1011 04:13:29.661867       1 binpack.go:183] resources [] record in weight but not found on any node
I1011 04:13:29.661878       1 binpack.go:167] Leaving binpack plugin. binpack.weight[1], binpack.cpu[1], binpack.memory[1], cpu[1], memory[1] ...
I1011 04:13:29.661887       1 enqueue.go:45] Enter Enqueue ...
I1011 04:13:29.661893       1 enqueue.go:63] Added Queue <child-queue-a> for Job <my-spark-operator-ha-apps-ns/spark-pg-1>
I1011 04:13:29.661937       1 enqueue.go:79] Try to enqueue PodGroup to 0 Queues
I1011 04:13:29.661948       1 enqueue.go:104] Leaving Enqueue ...
I1011 04:13:29.661956       1 allocate.go:77] Enter Allocate ...
I1011 04:13:29.661965       1 job_info.go:929] job spark-pg-1/my-spark-operator-ha-apps-ns actual: map[:8], ji.TaskMinAvailable: map[]
I1011 04:13:29.662052       1 allocate.go:132] Added Job <my-spark-operator-ha-apps-ns/spark-pg-1> into Queue <child-queue-a>
I1011 04:13:29.662063       1 allocate.go:96] Try to allocate resource to 1 Queues
I1011 04:13:29.662070       1 allocate.go:161] Try to allocate resource to Jobs in Queue <child-queue-a>
I1011 04:13:29.662076       1 allocate.go:161] Try to allocate resource to Jobs in Queue <child-queue-a>
I1011 04:13:29.662081       1 allocate.go:165] Can not find jobs for queue child-queue-a.
I1011 04:13:29.662086       1 allocate.go:98] Leaving Allocate ...
I1011 04:13:29.662093       1 reclaim.go:41] Enter Reclaim ...
I1011 04:13:29.662098       1 reclaim.go:50] There are <1> Jobs and <5> Queues in total for scheduling.
I1011 04:13:29.662105       1 job_info.go:929] job spark-pg-1/my-spark-operator-ha-apps-ns actual: map[:8], ji.TaskMinAvailable: map[]
I1011 04:13:29.662115       1 reclaim.go:67] Added Queue <child-queue-a> for Job <my-spark-operator-ha-apps-ns/spark-pg-1>
I1011 04:13:29.662125       1 reclaim.go:228] Leaving Reclaim ...
I1011 04:13:29.662131       1 preempt.go:54] Enter Preempt ...
I1011 04:13:29.662137       1 job_info.go:929] job spark-pg-1/my-spark-operator-ha-apps-ns actual: map[:8], ji.TaskMinAvailable: map[]
I1011 04:13:29.662146       1 preempt.go:78] Added Queue <child-queue-a> for Job <my-spark-operator-ha-apps-ns/spark-pg-1>
I1011 04:13:29.662153       1 preempt.go:108] No preemptors in Queue <child-queue-a>, break.
I1011 04:13:29.662160       1 preempt.go:221] Leaving Preempt ...
I1011 04:13:29.662166       1 backfill.go:53] Enter Backfill ...
I1011 04:13:29.662175       1 job_info.go:929] job spark-pg-1/my-spark-operator-ha-apps-ns actual: map[:8], ji.TaskMinAvailable: map[]
I1011 04:13:29.662184       1 backfill.go:104] Leaving Backfill ...
I1011 04:13:29.662299       1 session.go:297] Queue <queue-my-root> allocated resource keeps equal, no need to update queue status <map[cpu:{{16 0} {<nil>} 16 DecimalSI} memory:{{17179869184 0} {<nil>}  BinarySI} pods:{{8 0} {<nil>} 8 DecimalSI}]>.
I1011 04:13:29.662328       1 session.go:297] Queue <child-queue-a> allocated resource keeps equal, no need to update queue status <map[cpu:{{16 0} {<nil>} 16 DecimalSI} memory:{{17179869184 0} {<nil>}  BinarySI} pods:{{8 0} {<nil>} 8 DecimalSI}]>.
I1011 04:13:29.662342       1 session.go:297] Queue <child-queue-b> allocated resource keeps equal, no need to update queue status <map[cpu:{{0 0} {<nil>} 0 DecimalSI} memory:{{0 0} {<nil>} 0 DecimalSI}]>.
I1011 04:13:29.662376       1 session.go:319] Root queue deserved/guaranteed resource and allocated resource remains the same, no need to update the queue.
I1011 04:13:29.662386       1 session.go:297] Queue <default> allocated resource keeps equal, no need to update queue status <map[cpu:{{0 0} {<nil>} 0 DecimalSI} memory:{{0 0} {<nil>} 0 DecimalSI}]>.
I1011 04:13:29.662394       1 session.go:368] Close Session cc73d2e7-40bf-4f25-a9da-7977b7fbc3a0
I1011 04:13:29.662402       1 scheduler.go:128] End scheduling ...

3、submit second application

kubectl apply -f spark-pg-2.yaml
kubectl apply -f app3.yaml

result is

kubectl get all -n my-spark-operator-ha-apps-ns -o wide

NAME                         READY   STATUS    RESTARTS   AGE     IP             NODE                          NOMINATED NODE   READINESS GATES
pod/spark-pi-1-demo-driver   1/1     Running   0          3m      10.xxx.xxx.205   225   <none>           <none>
pod/spark-pi-1-demo-exec-1   1/1     Running   0          2m52s   10.xxx.xxx.180   224   <none>           <none>
pod/spark-pi-1-demo-exec-2   1/1     Running   0          2m52s   10.xxx.xxx.206   225   <none>           <none>
pod/spark-pi-1-demo-exec-3   1/1     Running   0          2m52s   10.xxx.xxx.179   224   <none>           <none>
pod/spark-pi-1-demo-exec-4   1/1     Running   0          2m52s   10.xxx.xxx.251   223   <none>           <none>
pod/spark-pi-1-demo-exec-5   1/1     Running   0          2m52s   10.xxx.xxx.207   225   <none>           <none>
pod/spark-pi-1-demo-exec-6   1/1     Running   0          2m51s   10.xxx.xxx.252   223   <none>           <none>
pod/spark-pi-1-demo-exec-7   1/1     Running   0          2m51s   10.xxx.xxx.181   224   <none>           <none>
pod/spark-pi-3-demo-driver   0/1     Pending   0          81s     <none>         <none>                        <none>           <none>




kubectl get podgroup -n my-spark-operator-ha-apps-ns spark-pg-1 -o yaml

apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"scheduling.volcano.sh/v1beta1","kind":"PodGroup","metadata":{"annotations":{"volcano.sh/preemptable":"true"},"name":"spark-pg-1","namespace":"my-spark-operator-ha-apps-ns"},"spec":{"minMember":1,"minResources":
{"cpu":"16","memory":"16Gi"},"queue":"child-queue-a"}}
    volcano.sh/preemptable: "true"
  creationTimestamp: "2025-10-11T04:12:22Z"
  generation: 6
  name: spark-pg-1
  namespace: my-spark-operator-ha-apps-ns
  resourceVersion: "51663130"
  uid: 0dd97664-807f-475d-84a4-4a34904b6db4
spec:
  minMember: 1
  minResources:
    cpu: "16"
    memory: 16Gi
  queue: child-queue-a
status:
  conditions:
  - lastTransitionTime: "2025-10-11T04:13:02Z"
    reason: tasks in gang are ready to be scheduled
    status: "True"
    transitionID: b4b68037-332e-446a-b7d5-386254e34ef1
    type: Scheduled
  phase: Running
  running: 8




kubectl get podgroup -n my-spark-operator-ha-apps-ns spark-pg-2 -o yaml

apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"scheduling.volcano.sh/v1beta1","kind":"PodGroup","metadata":{"annotations":{"volcano.sh/preemptable":"true"},"name":"spark-pg-2","namespace":"my-spark-operator-ha-apps-ns"},"spec":{"minMember":1,"minResources":
{"cpu":"16","memory":"16Gi"},"queue":"child-queue-b"}}
    volcano.sh/preemptable: "true"
  creationTimestamp: "2025-10-11T04:14:01Z"
  generation: 3
  name: spark-pg-2
  namespace: my-spark-operator-ha-apps-ns
  resourceVersion: "51663643"
  uid: a6927429-41c6-42b4-a634-20ae8f71b2cf
spec:
  minMember: 1
  minResources:
    cpu: "16"
    memory: 16Gi
  queue: child-queue-b
status:
  conditions:
  - lastTransitionTime: "2025-10-11T04:14:28Z"
    message: '1/1 tasks in gang unschedulable: pod group is not ready, 1 Pending,
      1 minAvailable; Pending: 1 Unschedulable'
    reason: NotEnoughResources
    status: "True"
    transitionID: bc2ca5cb-739a-4bf3-8af6-289b30116e7a
    type: Unschedulable
  phase: Pending

log is

I1011 04:14:57.751921       1 scheduler.go:101] Start scheduling ...
I1011 04:14:57.752052       1 node_info.go:229] imageStates is map[]
I1011 04:14:57.752178       1 node_info.go:229] imageStates is map[]
I1011 04:14:57.752329       1 node_info.go:229] imageStates is map[]
I1011 04:14:57.752406       1 node_info.go:229] imageStates is map[]
I1011 04:14:57.752453       1 node_info.go:229] imageStates is map[]
I1011 04:14:57.752494       1 node_info.go:229] imageStates is map[]
I1011 04:14:57.752535       1 cache.go:1391] The priority of job <my-spark-operator-ha-apps-ns/spark-pg-1> is </0>
I1011 04:14:57.752580       1 cache.go:1391] The priority of job <my-spark-operator-ha-apps-ns/spark-pg-2> is </0>
I1011 04:14:57.752611       1 cache.go:1427] "SnapShot for scheduling" jobNum=2 QueueNum=5 NodeNum=6
I1011 04:14:57.752633       1 cache.go:1430] "HyperNode snapShot for scheduling" tiers={} realNodesSet={} hyperNodesReadyToSchedule=false
I1011 04:14:57.752650       1 session.go:228] Add node 218 resources: cpu 16000.00, memory 32780976128.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00, pods 110.00, ephemeral-storage 85124969946000.00
I1011 04:14:57.752669       1 session.go:228] Add node 223 resources: cpu 16000.00, memory 32780976128.00, pods 110.00, ephemeral-storage 85124969946000.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00
I1011 04:14:57.752681       1 session.go:228] Add node 224 resources: cpu 16000.00, memory 32780976128.00, hugepages-512Mi 0.00, pods 110.00, ephemeral-storage 85124969946000.00, hugepages-2Mi 0.00
I1011 04:14:57.752690       1 session.go:228] Add node 225 resources: cpu 16000.00, memory 32780976128.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00, pods 110.00, ephemeral-storage 85124969946000.00
I1011 04:14:57.752699       1 session.go:228] Add node 222 resources: cpu 16000.00, memory 32780976128.00, pods 110.00, ephemeral-storage 85124969946000.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00
I1011 04:14:57.752710       1 session.go:228] Add node 219 resources: cpu 16000.00, memory 32780976128.00, ephemeral-storage 85124969946000.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00, pods 110.00
I1011 04:14:57.752725       1 session.go:234] Open Session bef78fa2-90d1-4a8f-9cc6-948e53021d02 with <2> Job and <5> Queues
I1011 04:14:57.753372       1 overcommit.go:75] Enter overcommit plugin ...
I1011 04:14:57.753394       1 overcommit.go:142] Leaving overcommit plugin.
I1011 04:14:57.753405       1 drf.go:185] Total Allocatable cpu 96000.00, memory 196685856768.00, ephemeral-storage 510749819676000.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00, pods 660.00
I1011 04:14:57.753485       1 factory.go:43] Register preBinder predicates successfully
I1011 04:14:57.753497       1 capacity.go:101] The total resource is <cpu 96000.00, memory 196685856768.00, pods 660.00, ephemeral-storage 510749819676000.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00>
I1011 04:14:57.753527       1 capacity.go:450] Considering Job <my-spark-operator-ha-apps-ns/spark-pg-1>.
I1011 04:14:57.753558       1 capacity.go:498] Queue child-queue-a allocated <cpu 16000.00, memory 17179869184.00, pods 8.00> request <cpu 16000.00, memory 17179869184.00, pods 8.00> inqueue <cpu 0.00, memory 0.00> elastic <cpu 0.00, memory 0.00, pods 8.00>
I1011 04:14:57.753564       1 capacity.go:450] Considering Job <my-spark-operator-ha-apps-ns/spark-pg-2>.
I1011 04:14:57.753578       1 capacity.go:498] Queue child-queue-b allocated <cpu 0.00, memory 0.00> request <cpu 2000.00, memory 2147483648.00, pods 1.00> inqueue <cpu 0.00, memory 0.00> elastic <cpu 0.00, memory 0.00>
I1011 04:14:57.753603       1 capacity.go:513] Successfully checked queue's hierarchical structure.
I1011 04:14:57.753613       1 capacity.go:521] The attributes of queue <queue-my-root> in capacity: deserved <cpu 30000.00, memory 32212254720.00>, realCapability <cpu 30000.00, memory 32212254720.00, pods 660.00, ephemeral-storage 510749819676000.00>, allocate <cpu 16000.00, memory 17179869184.00, pods 8.00>, request <cpu 18000.00, memory 19327352832.00, pods 9.00>, elastic <cpu 0.00, memory 0.00, pods 8.00>, share <0.53>
I1011 04:14:57.753634       1 capacity.go:521] The attributes of queue <child-queue-a> in capacity: deserved <cpu 12000.00, memory 12884901888.00>, realCapability <cpu 20000.00, memory 21474836480.00, pods 660.00, ephemeral-storage 510749819676000.00>, allocate <cpu 16000.00, memory 17179869184.00, pods 8.00>, request <cpu 16000.00, memory 17179869184.00, pods 8.00>, elastic <cpu 0.00, memory 0.00, pods 8.00>, share <1.33>
I1011 04:14:57.753653       1 capacity.go:521] The attributes of queue <child-queue-b> in capacity: deserved <cpu 18000.00, memory 19327352832.00>, realCapability <cpu 30000.00, memory 32212254720.00, pods 660.00, ephemeral-storage 510749819676000.00>, allocate <cpu 0.00, memory 0.00>, request <cpu 2000.00, memory 2147483648.00, pods 1.00>, elastic <cpu 0.00, memory 0.00>, share <0.00>
I1011 04:14:57.753668       1 capacity.go:521] The attributes of queue <root> in capacity: deserved <cpu 96000.00, memory 196685856768.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00, pods 660.00, ephemeral-storage 510749819676000.00>, realCapability <cpu 96000.00, memory 196685856768.00, pods 660.00, ephemeral-storage 510749819676000.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00>, allocate <cpu 16000.00, memory 17179869184.00, pods 8.00>, request <cpu 18000.00, memory 19327352832.00, pods 9.00>, elastic <cpu 0.00, memory 0.00, pods 8.00>, share <0.17>
I1011 04:14:57.753688       1 capacity.go:521] The attributes of queue <default> in capacity: deserved <cpu 0.00, memory 0.00>, realCapability <cpu 96000.00, memory 196685856768.00, ephemeral-storage 510749819676000.00, pods 660.00>, allocate <cpu 0.00, memory 0.00>, request <cpu 0.00, memory 0.00>, elastic <cpu 0.00, memory 0.00>, share <0.00>
I1011 04:14:57.753753       1 binpack.go:165] Enter binpack plugin ...
I1011 04:14:57.753761       1 binpack.go:183] resources [] record in weight but not found on any node
I1011 04:14:57.753771       1 binpack.go:167] Leaving binpack plugin. binpack.weight[1], binpack.cpu[1], binpack.memory[1], cpu[1], memory[1] ...
I1011 04:14:57.753780       1 enqueue.go:45] Enter Enqueue ...
I1011 04:14:57.753788       1 enqueue.go:63] Added Queue <child-queue-b> for Job <my-spark-operator-ha-apps-ns/spark-pg-2>
I1011 04:14:57.753798       1 enqueue.go:74] Added Job <my-spark-operator-ha-apps-ns/spark-pg-2> into Queue <child-queue-b>
I1011 04:14:57.753805       1 enqueue.go:63] Added Queue <child-queue-a> for Job <my-spark-operator-ha-apps-ns/spark-pg-1>
I1011 04:14:57.753813       1 enqueue.go:79] Try to enqueue PodGroup to 1 Queues
I1011 04:14:57.753823       1 overcommit.go:125] Sufficient resources, permit job <my-spark-operator-ha-apps-ns/spark-pg-2> to be inqueue
I1011 04:14:57.753838       1 capacity.go:767] job spark-pg-2 min resource <cpu 16000.00, memory 17179869184.00>, queue child-queue-b capability <cpu 30000.00, memory 32212254720.00, pods 660.00, ephemeral-storage 510749819676000.00> allocated <cpu 0.00, memory 0.00> inqueue <cpu 0.00, memory 0.00> elastic <cpu 0.00, memory 0.00>
I1011 04:14:57.753855       1 capacity.go:767] job spark-pg-2 min resource <cpu 16000.00, memory 17179869184.00>, queue queue-my-root capability <cpu 30000.00, memory 32212254720.00, pods 660.00, ephemeral-storage 510749819676000.00> allocated <cpu 16000.00, memory 17179869184.00, pods 8.00> inqueue <cpu 0.00, memory 0.00> elastic <cpu 0.00, memory 0.00, pods 8.00>
I1011 04:14:57.753873       1 capacity.go:767] job spark-pg-2 min resource <cpu 16000.00, memory 17179869184.00>, queue root capability <cpu 96000.00, memory 196685856768.00, pods 660.00, ephemeral-storage 510749819676000.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00> allocated <cpu 16000.00, memory 17179869184.00, pods 8.00> inqueue <cpu 0.00, memory 0.00> elastic <cpu 0.00, memory 0.00, pods 8.00>
I1011 04:14:57.753905       1 enqueue.go:104] Leaving Enqueue ...
I1011 04:14:57.753914       1 allocate.go:77] Enter Allocate ...
I1011 04:14:57.753924       1 job_info.go:929] job spark-pg-1/my-spark-operator-ha-apps-ns actual: map[:8], ji.TaskMinAvailable: map[]
I1011 04:14:57.753940       1 allocate.go:132] Added Job <my-spark-operator-ha-apps-ns/spark-pg-1> into Queue <child-queue-a>
I1011 04:14:57.753945       1 allocate.go:106] Job <my-spark-operator-ha-apps-ns/spark-pg-2> Queue <child-queue-b> skip allocate, reason: job status is pending.
I1011 04:14:57.753951       1 allocate.go:96] Try to allocate resource to 1 Queues
I1011 04:14:57.753956       1 allocate.go:161] Try to allocate resource to Jobs in Queue <child-queue-a>
I1011 04:14:57.753961       1 allocate.go:161] Try to allocate resource to Jobs in Queue <child-queue-a>
I1011 04:14:57.753966       1 allocate.go:165] Can not find jobs for queue child-queue-a.
I1011 04:14:57.753972       1 allocate.go:98] Leaving Allocate ...
I1011 04:14:57.753978       1 reclaim.go:41] Enter Reclaim ...
I1011 04:14:57.753983       1 reclaim.go:50] There are <2> Jobs and <5> Queues in total for scheduling.
I1011 04:14:57.753989       1 job_info.go:929] job spark-pg-1/my-spark-operator-ha-apps-ns actual: map[:8], ji.TaskMinAvailable: map[]
I1011 04:14:57.753998       1 reclaim.go:67] Added Queue <child-queue-a> for Job <my-spark-operator-ha-apps-ns/spark-pg-1>
I1011 04:14:57.754008       1 reclaim.go:228] Leaving Reclaim ...
I1011 04:14:57.754013       1 preempt.go:54] Enter Preempt ...
I1011 04:14:57.754019       1 job_info.go:929] job spark-pg-1/my-spark-operator-ha-apps-ns actual: map[:8], ji.TaskMinAvailable: map[]
I1011 04:14:57.754027       1 preempt.go:78] Added Queue <child-queue-a> for Job <my-spark-operator-ha-apps-ns/spark-pg-1>
I1011 04:14:57.754053       1 preempt.go:108] No preemptors in Queue <child-queue-a>, break.
I1011 04:14:57.754061       1 preempt.go:221] Leaving Preempt ...
I1011 04:14:57.754068       1 backfill.go:53] Enter Backfill ...
I1011 04:14:57.754074       1 job_info.go:929] job spark-pg-1/my-spark-operator-ha-apps-ns actual: map[:8], ji.TaskMinAvailable: map[]
I1011 04:14:57.754083       1 backfill.go:104] Leaving Backfill ...
I1011 04:14:57.754204       1 cache.go:1010] task unscheduleable my-spark-operator-ha-apps-ns/spark-pi-3-demo-driver, message: 1/1 tasks in gang unschedulable: pod group is not ready, 1 Pending, 1 minAvailable; Pending: 1 Unschedulable, skip by no condition update
I1011 04:14:57.754503       1 session.go:319] Root queue deserved/guaranteed resource and allocated resource remains the same, no need to update the queue.
I1011 04:14:57.754521       1 session.go:297] Queue <default> allocated resource keeps equal, no need to update queue status <map[cpu:{{0 0} {<nil>} 0 DecimalSI} memory:{{0 0} {<nil>} 0 DecimalSI}]>.
I1011 04:14:57.754544       1 session.go:297] Queue <queue-my-root> allocated resource keeps equal, no need to update queue status <map[cpu:{{16 0} {<nil>} 16 DecimalSI} memory:{{17179869184 0} {<nil>}  BinarySI} pods:{{8 0} {<nil>} 8 DecimalSI}]>.
I1011 04:14:57.754561       1 session.go:297] Queue <child-queue-a> allocated resource keeps equal, no need to update queue status <map[cpu:{{16 0} {<nil>} 16 DecimalSI} memory:{{17179869184 0} {<nil>}  BinarySI} pods:{{8 0} {<nil>} 8 DecimalSI}]>.
I1011 04:14:57.754576       1 session.go:297] Queue <child-queue-b> allocated resource keeps equal, no need to update queue status <map[cpu:{{0 0} {<nil>} 0 DecimalSI} memory:{{0 0} {<nil>} 0 DecimalSI}]>.
I1011 04:14:57.754585       1 session.go:368] Close Session bef78fa2-90d1-4a8f-9cc6-948e53021d02
I1011 04:14:57.754594       1 scheduler.go:128] End scheduling ...

[Johnny539]

according log segement，know spark-pg-2 enqueue reject by capacity plugin

I1011 04:14:57.753838       1 capacity.go:767] job spark-pg-2 min resource <cpu 16000.00, memory 17179869184.00>, queue child-queue-b capability <cpu 30000.00, memory 32212254720.00, pods 660.00, ephemeral-storage 510749819676000.00> allocated <cpu 0.00, memory 0.00> inqueue <cpu 0.00, memory 0.00> elastic <cpu 0.00, memory 0.00>
I1011 04:14:57.753855       1 capacity.go:767] job spark-pg-2 min resource <cpu 16000.00, memory 17179869184.00>, queue queue-my-root capability <cpu 30000.00, memory 32212254720.00, pods 660.00, ephemeral-storage 510749819676000.00> allocated <cpu 16000.00, memory 17179869184.00, pods 8.00> inqueue <cpu 0.00, memory 0.00> elastic <cpu 0.00, memory 0.00, pods 8.00>
I1011 04:14:57.753873       1 capacity.go:767] job spark-pg-2 min resource <cpu 16000.00, memory 17179869184.00>, queue root capability <cpu 96000.00, memory 196685856768.00, pods 660.00, ephemeral-storage 510749819676000.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00> allocated <cpu 16000.00, memory 17179869184.00, pods 8.00> inqueue <cpu 0.00, memory 0.00> elastic <cpu 0.00, memory 0.00, pods 8.00>

and then spark-pg-2 cannot trigger reclaim

I1011 04:13:29.662093       1 reclaim.go:41] Enter Reclaim ...
I1011 04:13:29.662098       1 reclaim.go:50] There are <1> Jobs and <5> Queues in total for scheduling.
I1011 04:13:29.662105       1 job_info.go:929] job spark-pg-1/my-spark-operator-ha-apps-ns actual: map[:8], ji.TaskMinAvailable: map[]
I1011 04:13:29.662115       1 reclaim.go:67] Added Queue <child-queue-a> for Job <my-spark-operator-ha-apps-ns/spark-pg-1>
I1011 04:13:29.662125       1 reclaim.go:228] Leaving Reclaim ...

try to make enqueue permit

      - name: capacity 
       ...
        enableJobEnqueued: false

[Jylx]

Thank you for your suggestions very much.
According to your request, I have obtained the following results.
the result does not satisfied Gang scheduling, and cannot reclaim.

1、make enqueue permit

kubectl -n volcano-system edit cm volcano-scheduler-configmap

volcano-scheduler.conf: |
    actions: "enqueue, allocate, reclaim, preempt, backfill"
    tiers:
    - plugins:
      - name: priority
      - name: gang
        enablePreemptable: false
      - name: conformance
    - plugins:
      - name: overcommit
      - name: drf
        enablePreemptable: false
      - name: predicates
      - name: capacity
        enableJobEnqueued: false
        enabledAllocatable: true
        enablePreemptive: true
        enableReclaimable: true
        enableQueueOrder: true
        enableHierarchy: true
      - name: nodeorder
      - name: binpack

kubectl -n volcano-system rollout restart deployment volcano-scheduler

2、create the two podgroup

kubectl apply -f spark-pg-2.yaml
kubectl apply -f spark-pg-3.yaml

3、submit first application

kubectl apply -f app1.yaml

the log is
https://docs.google.com/document/d/1RHS_19hBrZMkVkTripAwOjQOFsuE9w6HQ4JGu59Un5I/edit?usp=sharing
tab: logs-before

4、submit second application
kubectl apply -f app3.yaml

4.1 the result is

kubectl get all -n my-spark-operator-ha-apps-ns -o wide

NAME                         READY   STATUS    RESTARTS   AGE    IP             NODE                          NOMINATED NODE   READINESS GATES
pod/spark-pi-1-demo-driver   1/1     Running   0          114s   10.xxx.xxx.225   218   <none>           <none>
pod/spark-pi-1-demo-exec-1   1/1     Running   0          87s    10.xxx.xxx.32    219   <none>           <none>
pod/spark-pi-1-demo-exec-2   1/1     Running   0          87s    10.xxx.xxx.226   218   <none>           <none>
pod/spark-pi-1-demo-exec-3   1/1     Running   0          86s    10.xxx.xxx.247   224   <none>           <none>
pod/spark-pi-1-demo-exec-4   1/1     Running   0          86s    10.xxx.xxx.33    219   <none>           <none>
pod/spark-pi-1-demo-exec-5   1/1     Running   0          86s    10.xxx.xxx.227   218   <none>           <none>
pod/spark-pi-1-demo-exec-6   1/1     Running   0          85s    10.xxx.xxx.248   224   <none>           <none>
pod/spark-pi-1-demo-exec-7   1/1     Running   0          85s    10.xxx.xxx.34    219   <none>           <none>
pod/spark-pi-3-demo-driver   1/1     Running   0          33s    10.xxx.xxx.228   218   <none>           <none>
pod/spark-pi-3-demo-exec-1   1/1     Running   0          25s    10.xxx.xxx.250   224   <none>           <none>
pod/spark-pi-3-demo-exec-2   1/1     Running   0          25s    10.xxx.xxx.229   218   <none>           <none>
pod/spark-pi-3-demo-exec-3   1/1     Running   0          25s    10.xxx.xxx.35    219   <none>           <none>
pod/spark-pi-3-demo-exec-4   1/1     Running   0          25s    10.xxx.xxx.249   224   <none>           <none>
pod/spark-pi-3-demo-exec-5   1/1     Running   0          25s    10.xxx.xxx.36    219   <none>           <none>
pod/spark-pi-3-demo-exec-6   1/1     Running   0          24s    10.xxx.xxx.230   218   <none>           <none>
pod/spark-pi-3-demo-exec-7   0/1     Pending   0          24s    <none>         <none>                        <none>           <none>



kubectl get podgroups.scheduling.volcano.sh -n my-spark-operator-ha-apps-ns

NAME         STATUS    MINMEMBER   RUNNINGS   AGE
spark-pg-1   Running   1           8          3m53s
spark-pg-2   Running   1           7          3m50s


kubectl get podgroup -n my-spark-operator-ha-apps-ns spark-pg-1 -o yaml

apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"scheduling.volcano.sh/v1beta1","kind":"PodGroup","metadata":{"annotations":{"volcano.sh/preemptable":"true"},"name":"spark-pg-1","namespace":"my-spark-operator-ha-apps-ns"},"spec":{"minMember":1,"minResources":
{"cpu":"16","memory":"16Gi"},"queue":"child-queue-a"}}
    volcano.sh/preemptable: "true"
  creationTimestamp: "2025-10-11T11:28:07Z"
  generation: 8
  name: spark-pg-1
  namespace: my-spark-operator-ha-apps-ns
  resourceVersion: "51811287"
  uid: c63de7c0-8658-4e8d-bfa6-391c9eca2896
spec:
  minMember: 1
  minResources:
    cpu: "16"
    memory: 16Gi
  queue: child-queue-a
status:
  conditions:
  - lastTransitionTime: "2025-10-11T11:29:23Z"
    reason: tasks in gang are ready to be scheduled
    status: "True"
    transitionID: 0e6b1dd6-f679-47b1-a497-7e693ceed949
    type: Scheduled
  phase: Running
  running: 8



kubectl get podgroup -n my-spark-operator-ha-apps-ns spark-pg-2 -o yaml

apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"scheduling.volcano.sh/v1beta1","kind":"PodGroup","metadata":{"annotations":{"volcano.sh/preemptable":"true"},"name":"spark-pg-2","namespace":"my-spark-operator-ha-apps-ns"},"spec":{"minMember":1,"minResources":
{"cpu":"16","memory":"16Gi"},"queue":"child-queue-b"}}
    volcano.sh/preemptable: "true"
  creationTimestamp: "2025-10-11T11:28:10Z"
  generation: 6
  name: spark-pg-2
  namespace: my-spark-operator-ha-apps-ns
  resourceVersion: "51811715"
  uid: 1c557174-51b1-4c4e-aa99-049c82014794
spec:
  minMember: 1
  minResources:
    cpu: "16"
    memory: 16Gi
  queue: child-queue-b
status:
  conditions:
  - lastTransitionTime: "2025-10-11T11:30:22Z"
    reason: tasks in gang are ready to be scheduled
    status: "True"
    transitionID: eb573106-fcdf-409e-a3f1-942b4a02700b
    type: Scheduled
  phase: Running
  running: 7



kubectl get queue child-queue-a -o yaml

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"scheduling.volcano.sh/v1beta1","kind":"Queue","metadata":{"annotations":{},"name":"child-queue-a"},"spec":{"capability":{"cpu":"20","memory":"20Gi"},"deserved":{"cpu":12,"memory":"12Gi"},"parent":"queue-my-root
","reclaimable":true}}
  creationTimestamp: "2025-10-11T11:06:59Z"
  generation: 1
  name: child-queue-a
  resourceVersion: "51811224"
  uid: 8d3510e8-a6ec-487f-ad0f-11e4d6ad9a1c
spec:
  capability:
    cpu: "20"
    memory: 20Gi
  deserved:
    cpu: 12
    memory: 12Gi
  parent: queue-my-root
  reclaimable: true
  weight: 1
status:
  allocated:
    cpu: "16"
    memory: 16Gi
    pods: "8"
  reservation: {}
  state: Open



kubectl get queue child-queue-b -o yaml

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"scheduling.volcano.sh/v1beta1","kind":"Queue","metadata":{"annotations":{},"name":"child-queue-b"},"spec":{"capability":{"cpu":"30","memory":"30Gi"},"deserved":{"cpu":18,"memory":"18Gi"},"parent":"queue-my-root
","reclaimable":true}}
  creationTimestamp: "2025-10-11T11:06:59Z"
  generation: 1
  name: child-queue-b
  resourceVersion: "51811661"
  uid: a6483e8a-f93d-4880-a44e-4a0d2485712b
spec:
  capability:
    cpu: "30"
    memory: 30Gi
  deserved:
    cpu: 18
    memory: 18Gi
  parent: queue-my-root
  reclaimable: true
  weight: 1
status:
  allocated:
    cpu: "14"
    memory: 14Gi
    pods: "7"
  reservation: {}
  state: Open

4.2 the log is
https://docs.google.com/document/d/1RHS_19hBrZMkVkTripAwOjQOFsuE9w6HQ4JGu59Un5I/edit?usp=sharing
tab: logs-after

[Jylx]

I find the following text from https://volcano.sh/en/docs/actions/ .

Note: There is a conflict between enqueue action and preempt/reclaim action. If both enqueue action and preempt/reclaim action are configured, and enqueue action determines that the job cannot be queued, it may result in failure to generate Pending state Pods, thus failing to trigger preempt/reclaim action.

Is there a conflict between Gang Scheduler and Reclaim between queues for the Spark task？

[Johnny539]

gang check job ready depends on minMember, not minResources

func (ji *JobInfo) IsReady() bool {
	return ji.ReadyTaskNum()+ji.PendingBestEffortTaskNum() >= ji.MinAvailable
}

so maybe you should set podgroup spark-pg-2 minMember to 8

[Johnny539]

queue

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: queue-my-root
spec:
  capability:
    cpu: "30"
    memory: 30Gi
  deserved:
    cpu: 30
    memory: 30Gi
  parent: root
  reclaimable: true
  weight: 1
---
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: child-queue-a
spec:
  capability:
    cpu: "20"
    memory: 20Gi
  deserved:
    cpu: 12
    memory: 12Gi
  parent: queue-my-root
  reclaimable: true
  weight: 1
---
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: child-queue-b
spec:
  capability:
    cpu: "30"
    memory: 30Gi
  deserved:
    cpu: 18
    memory: 18Gi
  parent: queue-my-root
  reclaimable: true
  weight: 1

vcjob

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: pg-1
spec:
  minAvailable: 8
  schedulerName: volcano
  queue: child-queue-a
  tasks:
    - replicas: 7
      name: "worker"
      template:
        metadata:
          name: worker
        spec:
          containers:
            - image: nginx
              imagePullPolicy: IfNotPresent
              name: nginx
              resources:
                requests:
                  cpu: "2"
                  memory: "2Gi"
          restartPolicy: OnFailure
    - replicas: 1
      name: "driver"
      template:
        metadata:
          name: driver
        spec:
          containers:
            - image: nginx
              imagePullPolicy: IfNotPresent
              name: nginx
              resources:
                requests:
                  cpu: "2"
                  memory: "2Gi"
          restartPolicy: OnFailure
---
apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: pg-2
spec:
  minAvailable: 8
  schedulerName: volcano
  queue: child-queue-b
  tasks:
    - replicas: 7
      name: "worker"
      template:
        metadata:
          name: worker
        spec:
          containers:
            - image: nginx
              imagePullPolicy: IfNotPresent
              name: nginx
              resources:
                requests:
                  cpu: "2"
                  memory: "2Gi"
          restartPolicy: OnFailure
    - replicas: 1
      name: "driver"
      template:
        metadata:
          name: driver
        spec:
          containers:
            - image: nginx
              imagePullPolicy: IfNotPresent
              name: nginx
              resources:
                requests:
                  cpu: "2"
                  memory: "2Gi"
          restartPolicy: OnFailure

apply vc-job1

pg-1-driver-0                      1/1     Running   0          3m29s
pg-1-worker-0                      1/1     Running   0          3m29s
pg-1-worker-1                      1/1     Running   0          3m29s
pg-1-worker-2                      1/1     Running   0          3m29s
pg-1-worker-3                      1/1     Running   0          3m29s
pg-1-worker-4                      1/1     Running   0          3m29s
pg-1-worker-5                      1/1     Running   0          3m29s
pg-1-worker-6                      1/1     Running   0          3m29s

apply vc-job2

pg-1-driver-0                      0/1     Pending   0          20s
pg-1-worker-0                      1/1     Running   0          3m59s
pg-1-worker-1                      1/1     Running   0          3m59s
pg-1-worker-2                      1/1     Running   0          3m59s
pg-1-worker-3                      1/1     Running   0          3m59s
pg-1-worker-4                      1/1     Running   0          3m59s
pg-1-worker-5                      1/1     Running   0          3m59s
pg-1-worker-6                      1/1     Running   0          3m59s
pg-2-driver-0                      1/1     Running   0          63s
pg-2-worker-0                      1/1     Running   0          63s
pg-2-worker-1                      1/1     Running   0          63s
pg-2-worker-2                      1/1     Running   0          63s
pg-2-worker-3                      1/1     Running   0          63s
pg-2-worker-4                      1/1     Running   0          63s
pg-2-worker-5                      1/1     Running   0          63s
pg-2-worker-6                      1/1     Running   0          63s

check scheduler logs

$ kubectl logs -f -n volcano-system volcano-scheduler-68587c5cc-49p2l |grep "Try to reclaim Task"
E1011 15:15:07.670792       1 reclaim.go:193] Try to reclaim Task <default/pg-1-driver-0> for Tasks <default/pg-2-worker-6>
E1011 15:15:40.312416       1 reclaim.go:193] Try to reclaim Task <default/pg-1-driver-0> for Tasks <default/pg-2-worker-6>

reclaim works

[Jylx]

gang check job ready depends on minMember, not minResources

func (ji *JobInfo) IsReady() bool {
	return ji.ReadyTaskNum()+ji.PendingBestEffortTaskNum() >= ji.MinAvailable
}

so maybe you should set podgroup spark-pg-2 minMember to 8

My friend, you are very enthusiastic.
Thank you for your suggestions very much.

1、edit the minMember to 8

cat spark-pg-2.yaml

apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
  name: spark-pg-2
  namespace: my-spark-operator-ha-apps-ns #指定ns
  annotations:
    volcano.sh/preemptable: "true"
spec:
  queue: child-queue-b
  #minMember: 1
  minMember: 8
  minResources:
    cpu: "16"
    memory: "16Gi"

2、the result is

kubectl get all -n my-spark-operator-ha-apps-ns -o wide

NAME                         READY   STATUS      RESTARTS   AGE     IP             NODE                          NOMINATED NODE   READINESS GATES
pod/spark-pi-1-demo-driver   0/1     Completed   0          6m19s   10.xxx.xxx.238   218   <none>           <none>
pod/spark-pi-3-demo-driver   0/1     Pending     0          4m6s    <none>         <none>                        <none>           <none>


kubectl get podgroup -n my-spark-operator-ha-apps-ns spark-pg-2 -o yaml

apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"scheduling.volcano.sh/v1beta1","kind":"PodGroup","metadata":{"annotations":{"volcano.sh/preemptable":"true"},"name":"spark-pg-2","namespace":"my-s
park-operator-ha-apps-ns"},"spec":{"minMember":8,"minResources":{"cpu":"16","memory":"16Gi"},"queue":"child-queue-b"}}
    volcano.sh/preemptable: "true"
  creationTimestamp: "2025-10-12T04:57:09Z"
  generation: 3
  name: spark-pg-2
  namespace: my-spark-operator-ha-apps-ns
  resourceVersion: "52125534"
  uid: 34fefa5e-affd-4d43-93eb-2d5f52ca8361
spec:
  minMember: 8
  minResources:
    cpu: "16"
    memory: 16Gi
  queue: child-queue-b
status:
  conditions:
  - lastTransitionTime: "2025-10-12T04:58:21Z"
    message: '8/1 tasks in gang unschedulable: pod group is not ready, 1 Pending,
      8 minAvailable; Pending: 1 Unschedulable'
    reason: NotEnoughResources
    status: "True"
    transitionID: 369495e3-d165-4a73-9d5f-65f7120f393d
    type: Unschedulable
  phase: Inqueue

3、the log is

I1012 05:02:42.546767       1 scheduler.go:101] Start scheduling ...
I1012 05:02:42.546902       1 node_info.go:229] imageStates is map[]
I1012 05:02:42.547038       1 node_info.go:229] imageStates is map[]
I1012 05:02:42.547140       1 node_info.go:229] imageStates is map[]
I1012 05:02:42.547206       1 node_info.go:229] imageStates is map[]
I1012 05:02:42.547258       1 node_info.go:229] imageStates is map[]
I1012 05:02:42.547335       1 node_info.go:229] imageStates is map[]
I1012 05:02:42.547393       1 cache.go:1391] The priority of job <my-spark-operator-ha-apps-ns/spark-pg-1> is </0>
I1012 05:02:42.547453       1 cache.go:1391] The priority of job <my-spark-operator-ha-apps-ns/spark-pg-2> is </0>
I1012 05:02:42.547563       1 cache.go:1427] "SnapShot for scheduling" jobNum=2 QueueNum=5 NodeNum=6
I1012 05:02:42.547600       1 cache.go:1430] "HyperNode snapShot for scheduling" tiers={} realNodesSet={} hyperNodesReadyToSchedule=false
I1012 05:02:42.547622       1 session.go:228] Add node 225 resources: cpu 16000.00, memory 32780976128.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00, pods 110.00, ephemeral-storage 85124969946000.00
I1012 05:02:42.547675       1 session.go:228] Add node 218 resources: cpu 16000.00, memory 32780976128.00, hugepages-512Mi 0.00, pods 110.00, ephemeral-storage 85124969946000.00, hugepages-2Mi 0.00
I1012 05:02:42.547688       1 session.go:228] Add node 224 resources: cpu 16000.00, memory 32780976128.00, hugepages-512Mi 0.00, pods 110.00, ephemeral-storage 85124969946000.00, hugepages-2Mi 0.00
I1012 05:02:42.547722       1 session.go:228] Add node 223 resources: cpu 16000.00, memory 32780976128.00, hugepages-512Mi 0.00, pods 110.00, ephemeral-storage 85124969946000.00, hugepages-2Mi 0.00
I1012 05:02:42.547737       1 session.go:228] Add node 222 resources: cpu 16000.00, memory 32780976128.00, hugepages-512Mi 0.00, pods 110.00, ephemeral-storage 85124969946000.00, hugepages-2Mi 0.00
I1012 05:02:42.547754       1 session.go:228] Add node 219 resources: cpu 16000.00, memory 32780976128.00, ephemeral-storage 85124969946000.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00, pods 110.00
I1012 05:02:42.547777       1 session.go:234] Open Session 6d997000-b88e-4da6-bfbd-1b10415a2f94 with <2> Job and <5> Queues
I1012 05:02:42.548332       1 overcommit.go:75] Enter overcommit plugin ...
I1012 05:02:42.548365       1 overcommit.go:142] Leaving overcommit plugin.
I1012 05:02:42.548379       1 drf.go:185] Total Allocatable cpu 96000.00, memory 196685856768.00, pods 660.00, ephemeral-storage 510749819676000.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00
I1012 05:02:42.548475       1 factory.go:43] Register preBinder predicates successfully
I1012 05:02:42.548486       1 capacity.go:101] The total resource is <cpu 96000.00, memory 196685856768.00, ephemeral-storage 510749819676000.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00, pods 660.00>
I1012 05:02:42.548515       1 capacity.go:450] Considering Job <my-spark-operator-ha-apps-ns/spark-pg-1>.
I1012 05:02:42.548532       1 capacity.go:498] Queue child-queue-a allocated <cpu 0.00, memory 0.00> request <cpu 0.00, memory 0.00> inqueue <cpu 0.00, memory 0.00> elastic <cpu 0.00, memory 0.00>
I1012 05:02:42.548544       1 capacity.go:450] Considering Job <my-spark-operator-ha-apps-ns/spark-pg-2>.
I1012 05:02:42.548563       1 capacity.go:498] Queue child-queue-b allocated <cpu 0.00, memory 0.00> request <cpu 2000.00, memory 2147483648.00, pods 1.00> inqueue <cpu 16000.00, memory 17179869184.00> elastic <cpu 0.00, memory 0.00>
I1012 05:02:42.548594       1 capacity.go:513] Successfully checked queue's hierarchical structure.
I1012 05:02:42.548611       1 capacity.go:521] The attributes of queue <queue-my-root> in capacity: deserved <cpu 30000.00, memory 32212254720.00>, realCapability <cpu 30000.00, memory 32212254720.00, pods 660.00, ephemeral-storage 510749819676000.00>, allocate <cpu 0.00, memory 0.00>, request <cpu 2000.00, memory 2147483648.00, pods 1.00>, elastic <cpu 0.00, memory 0.00>, share <0.00>
I1012 05:02:42.548631       1 capacity.go:521] The attributes of queue <child-queue-a> in capacity: deserved <cpu 12000.00, memory 12884901888.00>, realCapability <cpu 20000.00, memory 21474836480.00, ephemeral-storage 510749819676000.00, pods 660.00>, allocate <cpu 0.00, memory 0.00>, request <cpu 0.00, memory 0.00>, elastic <cpu 0.00, memory 0.00>, share <0.00>
I1012 05:02:42.548644       1 capacity.go:521] The attributes of queue <child-queue-b> in capacity: deserved <cpu 18000.00, memory 19327352832.00>, realCapability <cpu 30000.00, memory 32212254720.00, pods 660.00, ephemeral-storage 510749819676000.00>, allocate <cpu 0.00, memory 0.00>, request <cpu 2000.00, memory 2147483648.00, pods 1.00>, elastic <cpu 0.00, memory 0.00>, share <0.00>
I1012 05:02:42.548664       1 capacity.go:521] The attributes of queue <root> in capacity: deserved <cpu 96000.00, memory 196685856768.00, pods 660.00, ephemeral-storage 510749819676000.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00>, realCapability <cpu 96000.00, memory 196685856768.00, hugepages-2Mi 0.00, hugepages-512Mi 0.00, pods 660.00, ephemeral-storage 510749819676000.00>, allocate <cpu 0.00, memory 0.00>, request <cpu 2000.00, memory 2147483648.00, pods 1.00>, elastic <cpu 0.00, memory 0.00>, share <0.00>
I1012 05:02:42.548688       1 capacity.go:521] The attributes of queue <default> in capacity: deserved <cpu 0.00, memory 0.00>, realCapability <cpu 96000.00, memory 196685856768.00, ephemeral-storage 510749819676000.00, pods 660.00>, allocate <cpu 0.00, memory 0.00>, request <cpu 0.00, memory 0.00>, elastic <cpu 0.00, memory 0.00>, share <0.00>
I1012 05:02:42.548747       1 binpack.go:165] Enter binpack plugin ...
I1012 05:02:42.548759       1 binpack.go:183] resources [] record in weight but not found on any node
I1012 05:02:42.548771       1 binpack.go:167] Leaving binpack plugin. binpack.weight[1], binpack.cpu[1], binpack.memory[1], cpu[1], memory[1] ...
I1012 05:02:42.548785       1 enqueue.go:45] Enter Enqueue ...
I1012 05:02:42.548797       1 enqueue.go:63] Added Queue <child-queue-b> for Job <my-spark-operator-ha-apps-ns/spark-pg-2>
I1012 05:02:42.548812       1 enqueue.go:63] Added Queue <child-queue-a> for Job <my-spark-operator-ha-apps-ns/spark-pg-1>
I1012 05:02:42.548826       1 enqueue.go:79] Try to enqueue PodGroup to 0 Queues
I1012 05:02:42.548837       1 enqueue.go:104] Leaving Enqueue ...
I1012 05:02:42.548850       1 allocate.go:77] Enter Allocate ...
I1012 05:02:42.548861       1 job_info.go:929] job spark-pg-1/my-spark-operator-ha-apps-ns actual: map[:1], ji.TaskMinAvailable: map[]
I1012 05:02:42.548881       1 allocate.go:132] Added Job <my-spark-operator-ha-apps-ns/spark-pg-1> into Queue <child-queue-a>
I1012 05:02:42.548887       1 job_info.go:929] job spark-pg-2/my-spark-operator-ha-apps-ns actual: map[:1], ji.TaskMinAvailable: map[]
I1012 05:02:42.548895       1 allocate.go:117] Job <my-spark-operator-ha-apps-ns/spark-pg-2> Queue <child-queue-b> skip allocate, reason: NotEnoughTasks, message Not enough valid tasks for gang-scheduling, valid: 1, min: 8
I1012 05:02:42.548901       1 allocate.go:96] Try to allocate resource to 1 Queues
I1012 05:02:42.548907       1 allocate.go:161] Try to allocate resource to Jobs in Queue <child-queue-a>
I1012 05:02:42.548912       1 allocate.go:161] Try to allocate resource to Jobs in Queue <child-queue-a>
I1012 05:02:42.548918       1 allocate.go:165] Can not find jobs for queue child-queue-a.
I1012 05:02:42.548928       1 allocate.go:98] Leaving Allocate ...
I1012 05:02:42.548935       1 reclaim.go:41] Enter Reclaim ...
I1012 05:02:42.548941       1 reclaim.go:50] There are <2> Jobs and <5> Queues in total for scheduling.
I1012 05:02:42.548947       1 job_info.go:929] job spark-pg-1/my-spark-operator-ha-apps-ns actual: map[:1], ji.TaskMinAvailable: map[]
I1012 05:02:42.548956       1 reclaim.go:67] Added Queue <child-queue-a> for Job <my-spark-operator-ha-apps-ns/spark-pg-1>
I1012 05:02:42.548963       1 job_info.go:929] job spark-pg-2/my-spark-operator-ha-apps-ns actual: map[:1], ji.TaskMinAvailable: map[]
I1012 05:02:42.548972       1 reclaim.go:59] Job <my-spark-operator-ha-apps-ns/spark-pg-2> Queue <child-queue-b> skip reclaim, reason: NotEnoughTasks, message Not enough valid tasks for gang-scheduling, valid: 1, min: 8
I1012 05:02:42.548982       1 reclaim.go:228] Leaving Reclaim ...
I1012 05:02:42.548988       1 preempt.go:54] Enter Preempt ...
I1012 05:02:42.548996       1 job_info.go:929] job spark-pg-1/my-spark-operator-ha-apps-ns actual: map[:1], ji.TaskMinAvailable: map[]
I1012 05:02:42.549003       1 preempt.go:78] Added Queue <child-queue-a> for Job <my-spark-operator-ha-apps-ns/spark-pg-1>
I1012 05:02:42.549009       1 job_info.go:929] job spark-pg-2/my-spark-operator-ha-apps-ns actual: map[:1], ji.TaskMinAvailable: map[]
I1012 05:02:42.549031       1 preempt.go:71] Job <my-spark-operator-ha-apps-ns/spark-pg-2> Queue <child-queue-b> skip preemption, reason: NotEnoughTasks, message Not enough valid tasks for gang-scheduling, valid: 1, min: 8
I1012 05:02:42.549039       1 preempt.go:108] No preemptors in Queue <child-queue-a>, break.
I1012 05:02:42.549045       1 preempt.go:221] Leaving Preempt ...
I1012 05:02:42.549051       1 backfill.go:53] Enter Backfill ...
I1012 05:02:42.549057       1 job_info.go:929] job spark-pg-1/my-spark-operator-ha-apps-ns actual: map[:1], ji.TaskMinAvailable: map[]
I1012 05:02:42.549065       1 job_info.go:929] job spark-pg-2/my-spark-operator-ha-apps-ns actual: map[:1], ji.TaskMinAvailable: map[]
I1012 05:02:42.549072       1 backfill.go:119] Job <my-spark-operator-ha-apps-ns/spark-pg-2> Queue <child-queue-b> skip backfill, reason: NotEnoughTasks, message Not enough valid tasks for gang-scheduling, valid: 1, min: 8
I1012 05:02:42.549092       1 backfill.go:104] Leaving Backfill ...
I1012 05:02:42.549307       1 cache.go:1010] task unscheduleable my-spark-operator-ha-apps-ns/spark-pi-3-demo-driver, message: 8/1 tasks in gang unschedulable: pod group is not ready, 1 Pending, 8 minAvailable; Pending: 1 Unschedulable, skip by no condition update
I1012 05:02:42.549386       1 session.go:319] Root queue deserved/guaranteed resource and allocated resource remains the same, no need to update the queue.
I1012 05:02:42.549404       1 session.go:297] Queue <default> allocated resource keeps equal, no need to update queue status <map[cpu:{{0 0} {<nil>} 0 DecimalSI} memory:{{0 0} {<nil>} 0 DecimalSI}]>.
I1012 05:02:42.549435       1 session.go:297] Queue <queue-my-root> allocated resource keeps equal, no need to update queue status <map[cpu:{{0 0} {<nil>} 0 DecimalSI} memory:{{0 0} {<nil>} 0 DecimalSI}]>.
I1012 05:02:42.549456       1 session.go:297] Queue <child-queue-a> allocated resource keeps equal, no need to update queue status <map[cpu:{{0 0} {<nil>} 0 DecimalSI} memory:{{0 0} {<nil>} 0 DecimalSI}]>.
I1012 05:02:42.549472       1 session.go:297] Queue <child-queue-b> allocated resource keeps equal, no need to update queue status <map[cpu:{{0 0} {<nil>} 0 DecimalSI} memory:{{0 0} {<nil>} 0 DecimalSI}]>.
I1012 05:02:42.549480       1 session.go:368] Close Session 6d997000-b88e-4da6-bfbd-1b10415a2f94
I1012 05:02:42.549490       1 scheduler.go:128] End scheduling ...

4、if the task is nginx, reclaim works. but if the task is spark, reclaim not works.

5、spark task is different with nginx task
for spark application, the driver is start first, and then driver start the executors.
may be it is the reason why the spark application cannot running when set podgoup minMember > 1 ?