ResourceQuotas should be restored before Pods (and related objects)

[jacksgt]

Hello,

thank you for maintaining this excellent tool.

I would like to propose adding ResourceQuotas to the list of high-priority resources during a restore operation.

velero/pkg/cmd/server/server.go

Line 583 in cc32375

High priorities:

The reason is that otherwise we can end up in a situation where the ResourceQuota is violated. Consider the following scenario for example:

• a namespace has a resource quota of pods=10
• the namespace has a deployment which sets replicas=15
• only 10 pods will exist in the namespace (limited by the ResourceQuota)
• take a backup of the namespace with velero
• delete the namespace
• restore the namespace with velero
• we end up in a situation with more than 10 pods because Pods, ReplicaSets and Deployments are restored BEFORE the resource quota

(I have verified this behavior with velero v1.13.1 & Kubernetes 1.28.2)

I know that I can control this behavior with:

--restore-resource-priorities=namespaces,resourcequotas,persistentvolumeclaims,secrets,configmaps,serviceaccounts,limitranges

https://velero.io/docs/v1.13/restore-reference/#restore-order

However given that ResourceQuotas are a core Kubernetes resource, I think it makes sense that all velero users benefit from this enhancement.

I'm happy to submit a PR for it.

[reasonerjt]

@jacksgt thanks for raising this issue.

I didn't have a chance to verify the situation if the Deployments are created before the ResourceQuota, it will not force the Deployment to scale down?

But it's also possible that the Deployments are created BEFORE the ResourceQuota is created on the source cluster before the backup?

[jacksgt]

Hi @reasonerjt ,

I didn't have a chance to verify the situation if the Deployments are created before the ResourceQuota, it will not force the Deployment to scale down?

No, the ResourceQuotas only enforce how many objects of a particular kind may be created, but it does not scaled down any resources. For example, when you have a quota of zero pods, you can still create a Deployment and even a ReplicaSet, but the Pods won't be created.

But it's also possible that the Deployments are created BEFORE the ResourceQuota is created on the source cluster before the backup?

I don't understand the question. If you are asking what the order should be, I suggest the following:

	HighPriorities: []string{
		"customresourcedefinitions",
		"namespaces",
		"storageclasses",
		"volumesnapshotclass.snapshot.storage.k8s.io",
		"volumesnapshotcontents.snapshot.storage.k8s.io",
		"volumesnapshots.snapshot.storage.k8s.io",
		"datauploads.velero.io",
		"persistentvolumes",
		"persistentvolumeclaims",
		"serviceaccounts",
		"secrets",
		"configmaps",
                // restore quotas before pods so the quota limits can be applied to any restored pod
                "resourcequotas", // <--- NEW
		"limitranges",
		"pods",
		// we fully qualify replicasets.apps because prior to Kubernetes 1.16, replicasets also
		// existed in the extensions API group, but we back up replicasets from "apps" so we want
		// to ensure that we prioritize restoring from "apps" too, since this is how they're stored
		// in the backup.
		"replicasets.apps",
		"clusterclasses.cluster.x-k8s.io",
		"endpoints",
		"services",
	},

[reasonerjt]

I'm thinking about the scenario that a user created 2 pods in a namespace, after that a ResourceQuota with 1 pod was created. If ResourceQuota does not scale down or remove the existing pods, the 2 pods will remain even the quota is only 1.

Then the user takes a backup of the namespace and tries to restore, if velero creates the ResourceQuota before the pods, only one pod will be created successfully, which is not consistent with backup.

[sseago]

@reasonerjt The other challenge is that when restoring the pods, which pods fail to restore (due to quota issues) ends up being a function of the order that Velero happened to restore in -- and if there were attached volumes, this would also lead to failed data restore in certain cases (fs-backup, WFFC datamover). It sounds like in this particular user's scenario, that's what they want, but we can't guarantee that other users want the same. In this case, I think the existing workaround of setting the restore priority in cases where users want quotas to prevent full restore is probably the best solution.

[reasonerjt]

@reasonerjt The other challenge is that when restoring the pods, which pods fail to restore (due to quota issues) ends up being a function of the order that Velero happened to restore in -- and if there were attached volumes, this would also lead to failed data restore in certain cases (fs-backup, WFFC datamover). It sounds like in this particular user's scenario, that's what they want, but we can't guarantee that other users want the same. In this case, I think the existing workaround of setting the restore priority in cases where users want quotas to prevent full restore is probably the best solution.

+1