distributed provisioning: unset "selected-node" for nodes which have no driver running

[pohly]

When deploying external-provisioner alongside the CSI driver on each node, there is one problem: if the scheduler picks a node which has no driver instance, then the volume is stuck because the usual "no capacity -> reschedule" recovery is never triggered.

A custom scheduler extension and capacity tracking can minimize the risk, but cannot prevent this entirely.

Possible solutions:

• deploy the driver on all nodes, let it report "no capacity" on those were it has no resources -> works today, but creates overhead
• deploy a central provisioner together with a driver component that knows about nodes where the driver runs -> can be done today, but implies that CSI drivers must be made aware of Kubernetes
• do something similar inside external-provisioner, probably based on node labels (specify node selector for "driver not running" and handle those)

[pohly]

deploy a central provisioner together with a driver component that knows about nodes where the driver runs -> can be done today, but implies that CSI drivers must be made aware of Kubernetes

This isn't ideal because "provisioning" will be started by the central provisioner for all nodes and then must be made to fail for those which do have a driver, which will emit additional events.

[pohly]

do something similar inside external-provisioner, probably based on node labels (specify node selector for "driver not running" and handle those)

This is conceptually very similar to setting AllowedTopologies in the storage class which uses late binding. The volume scheduler will check that before selecting a node, right?

If so, then this is probably the right solution for this issue because it avoids the problem entirely.

There's a slight race (node has the right labels, is selected for a PVC, labels get removed, driver no longer runs -> PVC stuck), but that should be rare and can be documented as a caveat for admins.

[pohly]

This is conceptually very similar to setting AllowedTopologies in the storage class which uses late binding. The volume scheduler will check that before selecting a node, right?

Yes, it does: https://github.com/kubernetes/kubernetes/blob/ba5f5bea64a7b42265e581e8c7fe633336bec79a/pkg/controller/volume/scheduling/scheduler_binder.go#L873-L877

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[pohly]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[pohly]

/remove-lifecycle rotten