storage: correct source statistics for multi-replica clusters #32676
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
aljoscha
force-pushed
the
storage-correct-source-stats-for-multi-replica-clusters
branch
from
c8a3ad8 to
b50f321
Compare
|
@martykulma & @petrosagg how do we like this approach? |
|
Updated to also include the same implementation for sinks |
…date Before, we were using `SourceStatisticsUpdate` both to collect and transmit statistics from cluster to the controller _and_ to keep those statistics in the controller. We want to make changes to how the controller incorporates statistics updates from the clusters, so we need to now separate these. IMO, this in itself is already an improvement because `SourceStatisticsUpdate` has a lot of functionality that is not needed in the controller slash can be confusing when read in the context of the controller.
Before, with sources running on multi-replica clusters (which are not enabled in prod right now), we would overcount some metrics. Specifically, there a gauge-metrics which would work correctly and there are counter metrics where we would overcount. For example, each instance of the source running on each replica would report `messages_received`. Now, we treat counter metrics differently: for those metrics where all sources are reporting basically the same counter update we only take the updates from the "first" replica. For gauges we just use all of them, and the assumption is that these are largely the same on all replicas. The one counter metric that is treated different from the others is `updates_committed`: only one replica will manage to commit updates, so we have to make sure we take this counter in from all replicas, to capture all messages that have been committed. Implements MaterializeInc/database-issues#9010
Before, with sinks running on multi-replica clusters (which are not enabled in prod right now), we would overcount some metrics. Now, we treat counter metrics differently: for those metrics where all sinks are reporting basically the same counter update we only take the updates from the "first" replica. The counter metrics that are treated different from the others are the `*_committed` metrics: only one replica will manage to commit updates, so we have to make sure we take this counter in from all replicas, to capture all messages that have been committed. Implements MaterializeInc/database-issues#9015
aljoscha
force-pushed
the
storage-correct-source-stats-for-multi-replica-clusters
branch
from
d115035 to
11fef79
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Before, with sources running on multi-replica clusters (which are not
enabled in prod right now), we would overcount some metrics.
Specifically, there a gauge-metrics which would work correctly and there
are counter metrics where we would overcount. For example, each instance
of the source running on each replica would report
messages_received.Now, we treat counter metrics differently: for those metrics where all
sources are reporting basically the same counter update we only take the
updates from the "first" replica. For gauges we just use all of them,
and the assumption is that these are largely the same on all replicas.
The one counter metric that is treated different from the others is
updates_committed: only one replica will manage to commit updates, sowe have to make sure we take this counter in from all replicas, to
capture all messages that have been committed.
Implements https://github.com/MaterializeInc/database-issues/issues/9010
WIP: @def- I still need to add a test that has sources and multi-replica clusters.