Address scalability issue when Node Watcher is enabled

[xing-yang]

We have an issue #75 to change the code to only watch Pods and Nodes when the Node Watcher component is enabled. We still need to address the scalability issue when Node Watcher is enabled:

kubernetes/kubernetes#102452 (comment)

[xing-yang]

@NickrenREN I wonder if you've seen a similar issue in production.

[NickrenREN]

Node Watcher is a single instance controller, what is the scalability issue ?

[xing-yang]

@NickrenREN It affects the e2e tests. Details are in this issue: kubernetes/kubernetes#102452

By disabling the external-health-monitor, the failure went away.

[NickrenREN]

IIUC, the root cause of the scalability issue you mention is: Node Watcher watches PVCs, Nodes and Pods ?
I just don't understand the reason. k8s default scheduler also does the same thing.

[NickrenREN]

Watch is persistent connection, and Node Watcher is a single instance controller. Is this really the root cause ?

[NickrenREN]

I saw many API Throttlings, so maybe we can decrease the API call frequency ?

[xing-yang]

Watch is persistent connection, and Node Watcher is a single instance controller. Is this really the root cause ?

This needs more investigation. The observation is that the failure went away when external-health-monitor was disabled, came back again when it is enabled, and went away again when it was disabled.

I saw many API Throttlings, so maybe we can decrease the API call frequency ?

We could try that.

[NickrenREN]

This needs more investigation. The observation is that the failure went away when external-health-monitor was disabled, came back again when it is enabled, and went away again when it was disabled.

This indicates the controller causes the failure (API throttling ?), but i still don't think Watch is the root cause.

[xing-yang]

This indicates the controller causes the failure (API throttling ?), but i still don't think Watch is the root cause.

The external-health-monitor controller added more load to the API server which might have triggered those failures.

[NickrenREN]

The external-health-monitor controller added more load to the API server which might have triggered those failures.

I agree, so we can try to decrease the API call frequency first.

[sonasingh46]

I would like to work on this issue. Will start to look into it and understand.

[sonasingh46]

/assign

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale