plugin: Add new reconcile metrics #1360
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sharnoff
force-pushed
the
sharnoff/plugin-reconcile-metrics
branch
from
e4d8f3f to
02c4f80
Compare
Omrigan
approved these changes
May 7, 2025
sharnoff
force-pushed
the
sharnoff/plugin-init-metrics
branch
from
e8be52f to
c561790
Compare
Implemented with a new WaitingTimer abstraction, which we use to track the aggregate amount of time where the queue is non-empty. The change to .golangci.yml is in order to exempt prometheus.Opts from exhaustruct requirements.
sharnoff
force-pushed
the
sharnoff/plugin-reconcile-metrics
branch
from
02c4f80 to
d55af20
Compare
|
Realized this doesn't handle objects being retried with a delay correctly (if still waiting, they'd be incorrectly counted towards the queue being non-empty when technically they're not ready to be used yet). Need to fix that before merging. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There's two metrics added here:
Number of reconcile workers
This mirrors the existing
controller_runtime_max_concurrent_reconcilesmetric we have from neonvm-controller, which allows us to determine the fraction of total worker-time that we're using (a measure of saturation).Total duration with items in the queue
This is roughly analogous to the Linux kernel's CPU PSI metric — other metrics of saturation (using total time spent reconciling or total time in the queue) are useful, but they tend to be easy to misinterpret when the amount of saturation is very skewed across the duration between metric samples. So the idea here is to help get a more accurate picture.
Resolves neondatabase/cloud#27613.