How to monitor your kafka connectors

This a quick set up of kafka, kafka ui and kafka connect with a monitoring through Prometheus and Grafana using docker compose.

Architecture

Setup

Required software:

• Docker, Docker Compose.

Here we are using a mongo kafka sink connector. Jar is downloaded from here and put in the kafka-jars.

We are using jmx exporter to collect kafka connect metrics.

Metrics config and grafana dashboard are download from here.

References

See this article for more explanation.