Update assemblies and modules Docs for Metrics Reporter integration #11329
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fvaleri
reviewed
Apr 8, 2025
documentation/assemblies/metrics/assembly-metrics-config-files.adoc
Outdated
Show resolved
Hide resolved
documentation/modules/metrics/ref-prometheus-metrics-config.adoc
Outdated
Show resolved
Hide resolved
documentation/assemblies/metrics/assembly_metrics-prometheus-setup.adoc
Outdated
Show resolved
Hide resolved
documentation/modules/metrics/proc-jmx-exporter-metrics-kafka-deploy-options.adoc
Outdated
Show resolved
Hide resolved
documentation/modules/metrics/proc-strimzi-metrics-reporter-kafka-deploy-options.adoc
Outdated
Show resolved
Hide resolved
documentation/modules/metrics/proc-strimzi-metrics-reporter-kafka-deploy-options.adoc
Outdated
Show resolved
Hide resolved
documentation/modules/metrics/proc-strimzi-metrics-reporter-kafka-deploy-options.adoc
Outdated
Show resolved
Hide resolved
| spec: | ||
| kafka: | ||
| # ... | ||
| metricsConfig: <1> |
There was a problem hiding this comment.
There is no reference for <1>. Is it really needed?
documentation/modules/metrics/proc-strimzi-metrics-reporter-kafka-deploy-options.adoc
Outdated
Show resolved
Hide resolved
ppatierno
reviewed
Apr 8, 2025
| ├── kafka-cruise-control-metrics.yaml <14> | ||
| ├── kafka-metrics.yaml <15> | ||
| ├── kafka-mirror-maker-2-metrics.yaml <16> | ||
| └── oauth-metrics.yaml <17> |
There was a problem hiding this comment.
a couple of questions here:
- aren't we getting examples for all the other components (apart Kafka) for using the Strimzi Metrics reporter?
- wouldn't be better to put the Prometheus JMX exporter related example in a dedicated folder now that we have the
strimzi-metrics-reporterone?
There was a problem hiding this comment.
@ppatierno Is this the structure you're recommending?
├── metrics
| ├── prometheus JMX exporter
| └──── grafana-dashboards
| └──── kafka-bridge-metrics.yaml
| └──── kafka-connect-metrics.yaml
| └──── kafka-cruise-control-metrics.yaml
| └──── kafka-metrics.yaml
| └──── kafka-mirror-maker-2-metrics.yaml
| └──── oauth-metrics.yaml
| ├── strimzi-metrics-reporter
| └────grafana-dashboards
| └──── kafka-metrics.yaml
| └──── README.md
There was a problem hiding this comment.
Yeah that was the idea thanks @OwenCorrigan76
@fvaleri @PaulRMellor wdyt?
There was a problem hiding this comment.
LGTM, but the folder name should be prometheus-jmx-exporter and we need a similar README.md for consistency.
There was a problem hiding this comment.
Yes. I'll update this docs PR and then update the main integration PR. Thanks
OwenCorrigan76
force-pushed
the
Docs_for_integrate_metrics_reporter
branch
from
e5596ab to
9fe4d31
Compare
PaulRMellor
reviewed
Apr 10, 2025
| <8> PodMonitor definitions translated by the Prometheus Operator into jobs for the Prometheus server to be able to scrape metrics data directly from pods. | ||
| <9> Example Grafana dashboards to be used with Metrics Reporter for the Kafka component. | ||
| <10> Resource for deploying and configuring Kafka with Strimzi Metrics Reporter. | ||
| <11> Example Grafana dashboards for the different Strimzi components. |
There was a problem hiding this comment.
Suggested change
| <11> Example Grafana dashboards for the different Strimzi components. | |
| <11> Grafana dashboards for components using the JMX Reporter. |
Comment on lines
34
to
36
| * xref:assembly-metrics-setup-{context}[Set up and deploy Prometheus to expose metrics] | ||
| * xref:proc-metrics-kafka-deploy-options-{context}[Deploy Kafka Exporter to provide additional metrics] | ||
| * xref:proc-jmx-exporter-metrics-kafka-deploy-options-{context}[Deploy Kafka Exporter to provide additional metrics] | ||
| * xref:proc-metrics-grafana-dashboard-{context}[Use Grafana to present the Prometheus metrics] |
There was a problem hiding this comment.
I deleted this section in PR #11307
I don't think we need it here, so would revert the changes to this file
| <6> Alerting rules examples for use with Prometheus Alertmanager (deployed with Prometheus). | ||
| <7> Installation resource file for the Prometheus image. | ||
| <8> PodMonitor definitions translated by the Prometheus Operator into jobs for the Prometheus server to be able to scrape metrics data directly from pods. | ||
| <9> Example Grafana dashboards to be used with Metrics Reporter for the Kafka component. |
There was a problem hiding this comment.
Suggested change
| <9> Example Grafana dashboards to be used with Metrics Reporter for the Kafka component. | |
| <9> Grafana dashboards for components using the Strimzi Metrics Reporter. |
|
|
||
| xref:proc-metrics-reporter-kafka-deploy-options-{context}[Enabling Strimzi Metrics Reporter through configuration] | ||
|
|
||
| Once the type is determined, using Prometheus with Strimzi, requires the following: |
There was a problem hiding this comment.
Suggested change
| Once the type is determined, using Prometheus with Strimzi, requires the following: | |
| After enabling metrics, you can integrate with Prometheus: |
| @@ -12,16 +12,23 @@ To expose metrics in Prometheus format, you add configuration to a custom resour | |||
| You must also make sure that the metrics are scraped by your monitoring stack. | |||
| Prometheus and Prometheus Alertmanager are used in the examples provided by Strimzi, but you can use also other compatible tools. | |||
|
|
|||
| Using Prometheus with Strimzi, requires the following: | |||
| You can enable either Prometheus JMX Exporter or Strimzi Metrics Reporter to expose metrics: | |||
There was a problem hiding this comment.
Suggested change
| You can enable either Prometheus JMX Exporter or Strimzi Metrics Reporter to expose metrics: | |
| You can expose metrics using one of the following approaches: |
| It is a good starting point for trying Prometheus with Strimzi. | ||
|
|
||
| This procedure shows how to deploy example Prometheus metrics configuration in the `Kafka` resource. | ||
| The process is the same when deploying the example files for other resources. |
There was a problem hiding this comment.
Suggested change
| The process is the same when deploying the example files for other resources. |
There was a problem hiding this comment.
we include a note/cross-ref on how to configure the Kafka Exporter as well in the JMX procedure. Do we want it here too?
Or maybe that needs to be separated from both now, so we link from each of these procedures. Note taken
| @@ -29,7 +29,7 @@ When accessing a dashboard, you can use the `port-forward` command to forward tr | |||
| The name of the Grafana pod is different for each user. | |||
|
|
|||
| .Prerequisites | |||
| * xref:proc-metrics-kafka-deploy-options-{context}[Metrics are configured for the Kafka cluster resource] | |||
| * xref:proc-jmx-exporter-metrics-kafka-deploy-options-{context}[Metrics are configured for the Kafka cluster resource] | |||
There was a problem hiding this comment.
should we have links for both options here as an either/or?
|
|
||
| [id='ref-jmx-metrics-prometheus-metrics-config-{context}'] | ||
|
|
||
| = Example Prometheus Jmx Exporter metrics configuration |
There was a problem hiding this comment.
Suggested change
| = Example Prometheus Jmx Exporter metrics configuration | |
| = Example Prometheus JMX Exporter metrics configuration |
| A label is a name-value pair. | ||
| Relabeling is the process of writing a label dynamically. | ||
| For example, the value of a label may be derived from the name of a Kafka server and client ID. | ||
| Strimzi uses the {StrimziMetricsReporter} or the {JMXExporter} to expose metrics through an HTTP endpoint, which can be scraped by the Prometheus server. |
There was a problem hiding this comment.
I wonder if this file is worth having for two sentences?
Maybe we can just add to the intro of the example metrics file section?
| @@ -42,6 +42,7 @@ | |||
| //Monitoring links | |||
| :GrafanaHome: link:https://grafana.com/[Grafana Labs^] | |||
| :JMXExporter: link:https://github.com/prometheus/jmx_exporter[Prometheus JMX Exporter^] | |||
| :StrimziMetricsReporter: link:https://github.com/strimzi/metrics-reporter[Strimzi Metrics Reporter^] | |||
There was a problem hiding this comment.
do we want to link to this? We only use the link in one place. Maybe we can live without it?
There was a problem hiding this comment.
I think there's another place that we should be linking to this, I've added a comment there, so maybe we should keep this?
OwenCorrigan76
force-pushed
the
Docs_for_integrate_metrics_reporter
branch
from
99e88b1 to
8fa4e37
Compare
|
@OwenCorrigan76 should this PR be held until #11051 is ready? |
fvaleri
approved these changes
Apr 14, 2025
PaulRMellor
approved these changes
Apr 16, 2025
| <13> Metrics configuration that defines Prometheus JMX Exporter relabeling rules for Kafka. | ||
| <14> Metrics configuration that defines Prometheus JMX Exporter relabeling rules for MirrorMaker 2. | ||
| <15> Metrics configuration that defines Prometheus JMX Exporter relabeling rules for OAuth 2.0. | ||
| <10> Example Grafana dashboards with Metrics Reporter for the Kafka component. |
There was a problem hiding this comment.
Suggested change
| <10> Example Grafana dashboards with Metrics Reporter for the Kafka component. | |
| <10> Grafana dashboards for components using the Metrics Reporter. |
| ├── kafka-cruise-control-metrics.yaml <14> | ||
| ├── kafka-metrics.yaml <15> | ||
| ├── kafka-mirror-maker-2-metrics.yaml <16> | ||
| └── oauth-metrics.yaml <17> | ||
|
|
||
| -- | ||
| <1> Example Grafana dashboards for the different Strimzi components. |
There was a problem hiding this comment.
Suggested change
| <1> Example Grafana dashboards for the different Strimzi components. | |
| <1> Grafana dashboards for components using the JMX Exporter. |
| [id='proc-metrics-kafka-deploy-options-{context}'] | ||
| = Enabling Prometheus metrics through configuration | ||
| [id='proc-jmx-exporter-metrics-kafka-deploy-options-{context}'] | ||
| = Enabling Prometheus Jmx Exporter |
There was a problem hiding this comment.
Suggested change
| = Enabling Prometheus Jmx Exporter | |
| = Enabling Prometheus JMX Exporter |
| // metrics/assembly_metrics-kafka.adoc | ||
|
|
||
| [id='proc-metrics-reporter-kafka-deploy-options-{context}'] | ||
| = Enabling Strimzi Metrics Reporter metrics |
There was a problem hiding this comment.
do we need metrics on the end or can we match with the JMX Exporter title: "Enabling Strimzi Metrics Reporter "
Comment on lines
17
to
19
| xref:proc-jmx-exporter-metrics-kafka-deploy-options-{context}[Enabling Prometheus Jmx Exporter] | ||
|
|
||
| xref:proc-metrics-reporter-kafka-deploy-options-{context}[Enabling Strimzi Metrics Reporter] |
There was a problem hiding this comment.
Suggested change
| xref:proc-jmx-exporter-metrics-kafka-deploy-options-{context}[Enabling Prometheus Jmx Exporter] | |
| xref:proc-metrics-reporter-kafka-deploy-options-{context}[Enabling Strimzi Metrics Reporter] | |
| * xref:proc-jmx-exporter-metrics-kafka-deploy-options-{context}[Enabling Prometheus Jmx Exporter] | |
| * xref:proc-metrics-reporter-kafka-deploy-options-{context}[Enabling Strimzi Metrics Reporter] |
OwenCorrigan76
force-pushed
the
Docs_for_integrate_metrics_reporter
branch
from
8fa4e37 to
f0e747f
Compare
|
Thanks a mil @PaulRMellor. I've addressed those latest comments. Hopefully we can merge as soon as the integration PR is merged. |
katheris
reviewed
Apr 17, 2025
| Using Prometheus with Strimzi, requires the following: | ||
| You can expose metrics using one of the following approaches: | ||
|
|
||
| * xref:proc-jmx-exporter-metrics-kafka-deploy-options-{context}[Enabling Prometheus Jmx Exporter] |
There was a problem hiding this comment.
Suggested change
| * xref:proc-jmx-exporter-metrics-kafka-deploy-options-{context}[Enabling Prometheus Jmx Exporter] | |
| * xref:proc-jmx-exporter-metrics-kafka-deploy-options-{context}[Enabling Prometheus JMX Exporter] |
|
|
||
| NOTE: The Prometheus JMX Exporter collects Java Management Extensions (JMX) from Kafka components and converts them into Prometheus metrics. | ||
| You do not require `jmxOptions` configuration in the custom resource of the component for the Prometheus JMX Exporter to function. | ||
| `jmxOptions` is only required if you need direct access to JMX from Kafka components. | ||
|
|
||
| //Example Prometheus metrics files | ||
| include::../../modules/metrics/ref-prometheus-metrics-config.adoc[leveloffset=+1] | ||
| Strimzi uses the StrimziMetricsReporter or the {JMXExporter} to expose metrics through an HTTP endpoint, which can be scraped by the Prometheus server. |
There was a problem hiding this comment.
Should Strimzi Metrics Reporter also have a link?
Suggested change
| Strimzi uses the StrimziMetricsReporter or the {JMXExporter} to expose metrics through an HTTP endpoint, which can be scraped by the Prometheus server. | |
| Strimzi uses the {StrimziMetricsReporter} or the {JMXExporter} to expose metrics through an HTTP endpoint, which can be scraped by the Prometheus server. |
| @@ -42,6 +42,7 @@ | |||
| //Monitoring links | |||
| :GrafanaHome: link:https://grafana.com/[Grafana Labs^] | |||
| :JMXExporter: link:https://github.com/prometheus/jmx_exporter[Prometheus JMX Exporter^] | |||
| :StrimziMetricsReporter: link:https://github.com/strimzi/metrics-reporter[Strimzi Metrics Reporter^] | |||
There was a problem hiding this comment.
I think there's another place that we should be linking to this, I've added a comment there, so maybe we should keep this?
scholzj
reviewed
Apr 21, 2025
| <13> Metrics configuration that defines Prometheus JMX Exporter relabeling rules for Kafka. | ||
| <14> Metrics configuration that defines Prometheus JMX Exporter relabeling rules for MirrorMaker 2. | ||
| <15> Metrics configuration that defines Prometheus JMX Exporter relabeling rules for OAuth 2.0. | ||
| <10> Grafana dashboards for components using the Metrics Reporter. |
There was a problem hiding this comment.
Suggested change
| <10> Grafana dashboards for components using the Metrics Reporter. | |
| <10> Grafana dashboards for components using the Strimzi Metrics Reporter. |
Signed-off-by: OwenCorrigan76 <[email protected]>
Signed-off-by: OwenCorrigan76 <[email protected]>
Signed-off-by: OwenCorrigan76 <[email protected]>
Signed-off-by: OwenCorrigan76 <[email protected]>
Signed-off-by: OwenCorrigan76 <[email protected]>
Signed-off-by: OwenCorrigan76 <[email protected]>
OwenCorrigan76
force-pushed
the
Docs_for_integrate_metrics_reporter
branch
from
f0e747f to
f1ca3be
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Type of change
Description
With newly added support for the Strimzi Metrics Reporter to Kafka brokers/controllers components, this PR updates the assemblies and modules documents to reflect the changes in the way we can deploy metrics to the Kafka brokers and controllers.
Updates:
Structure for new content:
Related to #11051
Checklist
Please go through this checklist and make sure all applicable tasks have been done