Implement backoff for k8s watcher retries #506
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue number:
#478
Description of changes:
The
kube::runtime
provides us utilities for watching for events from Kuberneteswatch
APIs and updating an internal cache of state, which is much more efficient than polling the k8s API for changes.If the brupop CRD is not installed, these watchers end up in an extremely tight retry loop to begin watching.
This PR:
kube
to get access to some nice default backoff functionality that was added in a recent release.controller
sleep for a longer fixed interval if there is nothing to do (which is true if the CRD is not installed)Testing done:
Deployed the changes to a cluster without the CRD installed.
Prior to this change, the agent logs were, several times per second, spammed with messages like this:
After this change, these messages are clearly backed-off exponentially, with a short initial burst followed by longer waits of 15+ seconds.
The controller also emits this log every 10 seconds:
Terms of contribution:
By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.