Inference services (Deployments/StatefulSets) encounter blocks when integrating with Kueue.

[Panlq]

#6715 (comment)

I discovered this issue while integrating an existing inference service with Kueue. The label kueue.x-k8s.io/queue-name of the existing Deployment is empty, and once integrated with Kueue, a queue-name will be injected into this label. However, based on the current update validation logic, an error is triggered. Does this scenario align with the design?

Inference services are not one-time tasks. Some of the already running ones cannot be deleted or recreated. When we need to integrate them with Kueue for quota management, we will encounter admission check failures during the subsequent service upgrade. Let’s discuss whether it is necessary to support this scenario.

empty label --->> non-empty
skip validate when oldQueueName == ""

kueue/pkg/controller/jobs/deployment/deployment_webhook.go

Line 134 in 06cb511

allErrs = append(allErrs, apivalidation.ValidateImmutableField(newQueueName, oldQueueName, queueNameLabelPath)...)

func (wh *Webhook) ValidateUpdate(ctx context.Context, oldObj, newObj runtime.Object) (warnings admission.Warnings, err error) {
	oldDeployment := fromObject(oldObj)
	newDeployment := fromObject(newObj)

	log := ctrl.LoggerFrom(ctx).WithName("deployment-webhook")
	log.V(5).Info("Validating update")

	oldQueueName := jobframework.QueueNameForObject(oldDeployment.Object())
	newQueueName := jobframework.QueueNameForObject(newDeployment.Object())

	allErrs := jobframework.ValidateQueueName(newDeployment.Object())

	// Prevents updating the queue-name if at least one Pod is not suspended
	// or if the queue-name has been deleted.
	isSuspended := oldDeployment.Status.ReadyReplicas == 0 
	if !isSuspended || newQueueName == "" {
		allErrs = append(allErrs, apivalidation.ValidateImmutableField(newQueueName, oldQueueName, queueNameLabelPath)...)
	}
	if !isSuspended || jobframework.IsWorkloadPriorityClassNameEmpty(newDeployment.Object()) {
		allErrs = append(allErrs, jobframework.ValidateUpdateForWorkloadPriorityClassName(oldDeployment.Object(), newDeployment.Object())...)
	}
	return warnings, allErrs.ToAggregate()
}

[mimowo]

Copy-pasting my response from the other thread for the ease of reading / following:

I don't think it is called in the design [that setting of queue-name is validated against]. I would say it was rather a precaution to validate against it, so I'm open to setting the label. However, we would need to test manually if this works as expected.