Priority set by localqueue and immutable

[richardhenwood]

I am interested in using queues designed to share surplus resource. For example, I may bursty jobs from one namespace. A second namespace has less important work that is ok to be interrupted. Owners of the first namespace are happy to share their surplus resource when they not using it. A queue system can achieve this with two localqueues both directing at the same resource pool. Jobs entering localqueue-A have priority 1000, jobs entering localqueue-B have priority 500. If priority can be set by the localqueue (and immutable) I can build this simple and easy-to-reason-about system.

From my experience, this queue design pattern is well proven and fairly common in existing HPC environments. I am interested in people's thoughts on achieving this with Kueue today - or changes that might make it simple in future.

[mimowo]

I think what you are asking is a common ask which may be modeled in a couple ways depending on details.

As the starting point for the discussion I would suggest the baseline solution is the "classical preemption" in a cohort. Here, you have LQ "BestEffort" pointing to CQ "BestEffort", and LQ "Guaraneed" pointing to CQ "Guaranteed".

Then, the preemption rules could be:

1. CQ Best effort:

nominalQuota: 0
borrowingLimit: 1000000 # infinity
  preemption: 
    withinClusterQueue: Never
    reclaimWithinCohort: Never

2. CQ Guaranteed:

nominalQuota: 100
borrowingLimit: 0
preemption:
  withinClusterQueue: Never
  reclaimWithinCohort: Any

For more complex setups you may also consider FairSharing where the "weight" allows you to control the relative priority between ClusterQueues. However, before exploring FairSharing it would be good to know if this is the direction you would like to go.

On that occassion I would love to have more examples of such "go to setups" in the docs https://kueue.sigs.k8s.io/docs/tasks/manage/administer_cluster_quotas/, so hopefully we could use your setup to enrich the documentation which currently is focusing more on semantic on individual building blocks, rather than entire system modelling.

cc @gabesaba @mwysokin @amy who may also have some ideas

[mimowo]

Yet another approach is using the AdmissionFairSharing which also has the concept of the ClusterQueue fair sharing weight.

cc @PBundyra

[richardhenwood]

Thanks for these ideas and discussion. I will need to review the documentation to understand the terms and relationships i.e. it is not immediately obvious what behavior will result from this configuration. For me there is a high-level of comprehension required here and that makes it hard to reason about the system. Queues are intuitive for many users so I would really like to understand if simpler design is possible and if fixing localqueue priority facilitates such design.

What I would like to be able to do is communicate out that our system looks like this:

queue       |      priority    |       nodes   
------------|------------------|------------------     
urgent      |      1000        |     A, B, C   
normal-A    |         0        |     A, B, C
normal-B    |         0        |     D, E, F 
backfill    |     -1000        |     A, B, C, D, E, F 

[amy]

Please take a look at this first to see if this applies to you. What you described sounds very similar to what I've experienced: #7129 (reply in thread)

If it doesn't apply, and you want a global queue with global concepts of priority, lets look at this example. Here my assumption would be... that the platform operator has a very clear idea of what types of workloads are urgent vs. not and can actually gate access to each localQueue. (As a platform operator, I do not have this insight because our fleet is just too large and heterogeneous.)

Today, Kueue doesn't have a concept of setting things based off the localQueue or clusterQueue you submit to. Chatted with @kannon92, he linked #7029 as an example of setting a default based off the localQueue you submit to. So setting workload priority based off localQueue could possibly be a feature Kueue offers if it has enough interest.

[amy]

Re: On that occassion I would love to have more examples of such "go to setups" in the docs Agreed!

I plan to pick this one up soon: #6932

[amy]

At a higher level, you're describing having effectively global priority within a global queue for ordering. This causes a lot of issues for larger organizations with complicated team structures sharing the same cluster and there's resource contention. I'd be curious if for your example, there's a large organization backing it? Or if its literally 2 teams / 2 categories of workloads sharing 1 HPC environment.

[richardhenwood]

Thanks for the prompt response. For some more background: our users often their understanding of queues from their experience of them in academia. What I have observed is that the academic queuing systems are described in terms of multiple queues that provide different TTLs, job scales etc. Queues may also be used to segment jobs by cost, urgency. Examples include:

• hpc.mil apparently no control of job priority, they are granted access to a 'Urgent' queue instead.
• TACC/Stampede3 do not apparently allow users to modify job priority. The operators use priority to increase through-put, apparently.
• NMState apparently use priority value to implement a backfill queue.

What I have observed with our users where we have provided them with freedom to set priority is:

• they begin to set jobs 'high' routinely. This creates support requests: why isn't my job getting to run?
• they are thoughtful they negotiate to run their urgent jobs. This devalues the queuing system an adds manual management overhead.
• it is exceedingly hard to reason about job behavior with multiple queues, resourcepools, lent/borrowed nodes, and job priority.

After experiencing large job queuing systems that prohibit users setting priority, and observing these issues, I have become biased against users setting priority. Hence my discussion point: can I take away user control priority and make it a function of localqueue?

[amy]

they begin to set jobs 'high' routinely. This creates support requests: why isn't my job getting to run?

We experienced this exact problem painfully. To expand what you're saying, you have a global queue, with global priority, but between tenants they all want their jobs to run, so set their priority to highest. Then priority within the cluster becomes meaningless. For this model, you "oversubscribe" the cluster to some arbitrary percentage like 200%. And when the original goal was to increase cluster allocation, it actually becomes unmanageable for SRE teams because they get paged when P1 workloads don't run. This usually happens when someone submits a giant P1 workload and kicks everyone else out. So then the organization changes to not allowing oversubscription, but then the cluster allocation drops significantly because not everyone is using the quota they were vended.

I think you're still modeling quota globally. Instead of thinking about a global queue and global priority, for a sufficiently large organization perhaps this model could be useful:

• 1 ClusterQueue per project or tenant
• tenant scoped priority
• vend each tenant quota, when they're not using it, other tenants can use it.

So you have something like this:
tenant ClusterQueue 1: 5 units
tenant ClusterQueue 2: 5 units
tenant ClusterQueue 3: 5 units
All ClusterQueues in Cohort-1 - https://kueue.sigs.k8s.io/docs/concepts/cohort/
So the total cohort pool here is 15 units.

So here are some basic usage models:
Example 1:

1. tenant ClusterQueue 1: submits 11 units
2. tenant ClusterQueue 2: nothing
3. tenant ClusterQueue 3: nothing
   All 11 in tenant queue 1 can run. It is "bursting" 6 units past its vended 5 units of nominalQuota.

Example 2:

1. tenant ClusterQueue 1: submits 11 units
2. tenant ClusterQueue 2: submits 11 units
3. tenant ClusterQueue 3: submits 11 units
   5 will run per queue. 6 will pend per queue.

Let's focus on example 2:
Here, for the 5 running per queue, workload priority is localized to each tenant. They can order which workloads they want to run within their 5 units.

What I'm trying to convey is, I'm assuming its not just 1 tenant with high priority workloads. I would guess that high priority workloads are split across tenants. And each tenant ALSO has low priority workloads. The paradigm shift is moving away from splicing a global queue with global priority (you do this by having only 1 ClusterQueue). Then moving to: vend quota to each tenant (ie having 1 ClusterQueue per tenant), let them order their workload priority within their vended quota.