Add cluster-trace-collector tool #1247
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
No changes to the coverage.
HTML Report |
Basically, make there's the same set of images every time and they're in the same order. And if there's an image intentionally missing, comment why. Also, align spacing in the places where that's feasible, because it makes it easier to see the set of images.
This new tool exists so that we can capture a scheduling/scaling "trace" from production (basically: a record of exactly all the operations that happened). This way, we have some real data availalbe to use in simulations for changes to our scheduling policies, AND we can use that data to extrapolate out to how changes in load will be handled. The architecture is pretty simple: * Reuse pkg/util/watch to get a stream of events on Pod and Node objects * Extract the useful bits out of each event * If the useful bits changed, send it off to S3/Azure Blob with an event sink from pkg/reporting
sharnoff
force-pushed
the
sharnoff/cluster-trace
branch
from
6a2eccc to
7b004fe
Compare
sharnoff
added a commit
that referenced
this pull request
Mar 11, 2025
Basically, make there's the same set of images every time and they're in the same order. And if there's an image intentionally missing, comment why. Also, align spacing in the places where that's feasible, because it makes it easier to see the set of images. --- Noticed some painful merge conflicts trying to update #1247, figured this bit of cleanup would help (it does!).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This new tool exists so that we can capture a scheduling/scaling "trace" from production (basically: a record of exactly all the operations that happened). This way, we have some real data available to use in simulations for changes to our scheduling policies, AND we can use that data to extrapolate out to how changes in load will be handled.
The architecture is pretty simple:
pkg/util/watchto get a stream of events on Pod and Node objectspkg/reportingref neondatabase/cloud#23343
Notes: