distributed provisioning: PVC for other node not ignored silently

[pohly]

After a "resource exhausted" error and selecting a different node, the original provisioner instance keeps getting updates and/or continues to work on a PVC that it should ignore:

I0902 17:33:16.091273       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"example-app-jx8rr-data", UID:"f5008265-3c43-4385-a78d-4fa3f72fb611", APIVersion:"v1", ResourceVersion:"485045", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "csi-hostpath-fast": rpc error: code = ResourceExhausted desc = requested capacity 53687091200 exceeds remaining capacity for "fast", 100Gi out of 100Gi already used
I0902 17:33:16.092186       1 connection.go:185] GRPC response: {"maximum_volume_size":{},"minimum_volume_size":{}}
I0902 17:33:16.092316       1 connection.go:186] GRPC error: <nil>
I0902 17:33:16.092419       1 capacity.go:641] Capacity Controller: no need to update csisc-5mph5 for {segment:0xc000318e10 storageClassName:csi-hostpath-fast}, same capacity 0 and correct owner
I0902 17:33:16.097560       1 controller.go:1426] provision "default/example-app-jx8rr-data" class "csi-hostpath-fast": volume rescheduled because: failed to provision volume with StorageClass "csi-hostpath-fast": rpc error: code = ResourceExhausted desc = requested capacity 53687091200 exceeds remaining capacity for "fast", 100Gi out of 100Gi already used
I0902 17:33:16.097697       1 controller.go:1095] Stop provisioning, removing PVC f5008265-3c43-4385-a78d-4fa3f72fb611 from claims in progress
I0902 17:33:24.229631       1 controller.go:1332] provision "default/example-app-jx8rr-data" class "csi-hostpath-fast": started
W0902 17:33:24.229729       1 controller.go:958] Retrying syncing claim "f5008265-3c43-4385-a78d-4fa3f72fb611", failure 0
E0902 17:33:24.229782       1 controller.go:981] error syncing claim "f5008265-3c43-4385-a78d-4fa3f72fb611": failed to get target node: node "aks-workerpool-15818640-vmss00000a" not found
I0902 17:33:24.229839       1 controller.go:1332] provision "default/example-app-jx8rr-data" class "csi-hostpath-fast": started
W0902 17:33:24.229877       1 controller.go:958] Retrying syncing claim "f5008265-3c43-4385-a78d-4fa3f72fb611", failure 1
E0902 17:33:24.229890       1 controller.go:981] error syncing claim "f5008265-3c43-4385-a78d-4fa3f72fb611": failed to get target node: node "aks-workerpool-15818640-vmss00000a" not found

This instance ran on vmss000000. The "failed to get target node" error is for a different node which isn't in the static node info cache.

[pohly]

/assign

[pohly]

The error comes from the provisioner lib: it tries to retrieve the node object before asking the provisioner whether the PVC should be provisioned. We discussed whether the lib should do the node lookup and decided to continue doing it there, but in this case here that isn't suitable.

One possible solution is to enhance the lister so that it always returns fake node objects:

external-provisioner/cmd/csi-provisioner/csi-provisioner.go

Line 336 in d686130

nodeLister = nodes.Lister()

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[pohly]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[pohly]

/help
/remove-lifecycle stale

[k8s-ci-robot]

@pohly:
This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• Does this issue have zero to low barrier of entry?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/help
/remove-lifecycle stale

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.