Scheduling preemptible and non-preemptible workloads in a single ClusterQueue

[jenny-yang-ai]

Followup discussion on #6041

I'd like to extend Kueue by permitting scheduling preemptible and non-preemptible workloads in a single ClusterQueue. I have the following requirements for my use case:

1. Clusterqueues can borrow unused resources from other clusterqueues
2. If a workload is submitted to a clusterqueue whose quota is not fully used by itself, the workload should be admitted immediately. (Implied here is that a workload in a CQ borrowing resources may be preempted)
3. Workloads should be admitted BestFIFO with both preemptible and non-preemptible workloads having the same priority. Preemptible workloads may be admitted before non-preemptible ones if the resources the non-preemptible one would be using are at risk of being preempted.
4. Users can specify if a workload is preemptible or non-preemptible
5. Non-preemptible workloads should never be preempted.

I thought the idea extending WorkloadPriorityClass proposed by @mimowo was great. Fleshing it out further, here are the changes needed:

1. Add a field to WorkloadPriorityClass called PreemptionPolicy that is Always or Never (can be extended later)
2. In clusterqueue_snapshot.go, track NonPreemptibleUsage which calculates the current usage by non-preemptible workloads for the given FlavorResource.
3. In flavorassigner.go/fitsResourceQuota, if a workload is nonpreemptible, check non-preemptible quota constraint
4. In preemption.go/findCandidates, skip the nonpreemptible workloads
5. IsNonpreemptible is defined as getWorkloadPriorityClass(w).preemptionPolicy == Never

I'm making this post to both sanity check that this approach would satisfy my requirements and that this is the right design. In my deployment, I plan to create 2 WorkloadPriorityClass with the same value so that the current BestFIFO ordering I have isn't affected. Since Kueue will track the non-preemptible usage, req 2 should be satisfied because there wont be a non-preemptible workload borrowing from another CQ.

If this looks good, is the next step to open a PR with the change + edits to KEP-973?

[kannon92]

This makes sense to me.

I think you can go ahead and open up an update on the KEP. There may be design discussions on there but that is to be expected.

[jenny-yang-ai]

Opened the PR here #6124