So our Prometheus server is now able to scrape Kafka broker metrics. This metric is available to a jmx_exporter by default Grafana Dashboard for Kafka Brokers The metrics from JMX exporter can be queried like any other metrics. This would bring up something similar as below Query Prometheus using PromQL You can check if Prometheus server is able to scrape the metrics is by navigating to Prometheus UI on Prometheus can also show you if it is scrapping metrics of Kafka Brokers prometheus -config.file= "prometheus.yml" =400d = "data/" In my case it is $KAFKA_HOME/conf Step – 2 – Edit kafka-server-start.sh Step – 1 – Goto the Kafka configuration directory
Integrating Kafka and JMX exporter is easy and it requires only one line added! 😉 Here goes I downloaded and have stored my configuration for JMX exporter in the $KAFKA_HOME/config/jmx_exporter.yml Configure Kafka broker There is already a sample configuration file to get us started and it is available on this link. Naming and filtering of the metrics can be done via regex expressions as a configuration in a YAML file. Rules around the naming of those metrics.Which metrics to scrape and which “NOT” to scrape.To enable JMX exporter to scrape metrics it needs to know a few things Kafka exposes a lot of metrics and they are really well-documented here. Note: The steps for JMX exporter need to be performed on all the brokers. You just need to know which files to copy or modify files. This is probably the most important part of this blog. The usual place for this would be $KAFKA_HOME/libs Configure JMX exporter for Kafka Step 2 – Once downloaded the jar needs to be placed along with other Kafka jars. Wget https: // /remotecontent? filepath=io /prometheus /jmx /jmx_prometheus_ja Step 1 – JMX exporter can be downloaded easily via the maven repo. Grafana Dashboard for Kafka Lag exporter.
The blog is divided into the following sections For the purpose of this blog, we have a three-node cluster on AWS.
#KAFKA EXPORTER INSTALL#
In case, you want to install Kafka, head over to this link which has got very nice steps. This blog assumes that you have a working Kafka cluster. Kafka’s metrics are well documented and are available on this link. In this blog, we will use a combination of JMX exporter and a pre-built exporter to monitor Kafka. Till now we have used pre-built exporters for Linux, Docker and JMX exporter for Cassandra. In this blog entry, we will see how we can monitor the state of Kafka brokers and also how we can monitor Kafka topic lags. Monitoring Kafka using Prometheus is easy.
0 kommentar(er)
