Prometheus exporter for Kafka high level consumer and topic, broker details

[Nipun Talukdar]

Hi All,

I have added a Kafka consumer lag, offset and topic details monitor for Prometheus.

Please check it here https://github.com/nipuntalukdar/kafka_consumer_exporter

Thanks,

Nipun Talukdar

[Matthias Rampke]

Hi,

This looks really interesting, I'm going to try it out! Can you explain the difference to https://github.com/danielqsj/kafka_exporter ? When would I use one or the other?

Thank you!

/MR

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/ab688bc9-bcbf-493d-8095-3cb0bde846d6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nipun Talukdar]

I glanced through the source of the other kafka exporter and it really makes too many calls to Kafka for getting the stats. For example, to get the highest or newest offset of each partition, it makes a separate call. So, if we have 100 topics and each topic has 100 partitions, then at least 100 * 100 calls will be made to Kafka brokers to get the these details. With big Kafka clusters with 1000s of partitions, this will be really an issue. It will make another 100 * 100 calls to get the details of in-sync-replicas and  100 * 100 calls for replica details. 

While https://github.com/nipuntalukdar/kafka_consumer_exporter  groups the topics, partitions by brokers and bring and will make at most one call to each broker for these details. Error handling also seems to be better here. 

https://github.com/danielqsj/kafka_exporter/blob/master/kafka_exporter.go will go through each brokers to get the consumer group offsets, but actually it should make a call to the co-ordinator broker for a consumer group. Also, if any broker throws any exception or any broker is down, it just stops getting the consumer offsets and breaks out of the loop where it was getting the consumer offsets group by groups. 

https://github.com/nipuntalukdar/kafka_consumer_exporter will perform better even when makes many metrics collections calls around the same time, while https://github.com/danielqsj/kafka_exporter will not perform well and it will also tax the Kafka cluster with too many calls.  Right now https://github.com/nipuntalukdar/kafka_consumer_exporter  doesn't export in-sync-replica count for each partition, while https://github.com/danielqsj/kafka_exporter  exports them.  https://github.com/nipuntalukdar/kafka_consumer_exporter exports number of topics in  error, also number of partitions in error and in good state,  while https://github.com/danielqsj/kafka_exporter doesn't. Also, https://github.com/nipuntalukdar/kafka_consumer_exporter yet to have a dockerfile added, while https://github.com/danielqsj/kafka_exporter already has a dockerfile. 

On Monday, 13 November 2017 17:53:08 UTC+5:30, Matthias Rampke wrote:

Hi,

This looks really interesting, I'm going to try it out! Can you explain the difference to https://github.com/danielqsj/kafka_exporter ? When would I use one or the other?

Thank you!

/MR

On Sun, Nov 12, 2017 at 6:04 AM Nipun Talukdar <nipun.t...@gmail.com> wrote:

Hi All,

I have added a Kafka consumer lag, offset and topic details monitor for Prometheus.

Please check it here https://github.com/nipuntalukdar/kafka_consumer_exporter

Thanks,

Nipun Talukdar

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

[Matthias Rampke]

Thank you for the detailed write-up! Sounds like there are different aspects to each that the other does not fulfill.

We try to avoid duplication even between unofficial exporters as much as possible. Is there a chance you two could get together and create an exporter that combines the best of both?

/MR

On Tue, Nov 14, 2017, 04:00 Nipun Talukdar <nipun.t...@gmail.com> wrote:

I glanced through the source of the other kafka exporter and it really makes too many calls to Kafka for getting the stats. For example, to get the highest or newest offset of each partition, it makes a separate call. So, if we have 100 topics and each topic has 100 partitions, then at least 100 * 100 calls will be made to Kafka brokers to get the these details. With big Kafka clusters with 1000s of partitions, this will be really an issue. It will make another 100 * 100 calls to get the details of in-sync-replicas and  100 * 100 calls for replica details. 

While https://github.com/nipuntalukdar/kafka_consumer_exporter  groups the topics, partitions by brokers and bring and will make at most one call to each broker for these details. Error handling also seems to be better here. 

https://github.com/danielqsj/kafka_exporter/blob/master/kafka_exporter.go will go through each brokers to get the consumer group offsets, but actually it should make a call to the co-ordinator broker for a consumer group. Also, if any broker throws any exception or any broker is down, it just stops getting the consumer offsets and breaks out of the loop where it was getting the consumer offsets group by groups. 

https://github.com/nipuntalukdar/kafka_consumer_exporter will perform better even when makes many metrics collections calls around the same time, while https://github.com/danielqsj/kafka_exporter will not perform well and it will also tax the Kafka cluster with too many calls.  Right now https://github.com/nipuntalukdar/kafka_consumer_exporter  doesn't export in-sync-replica count for each partition, while https://github.com/danielqsj/kafka_exporter  exports them.  https://github.com/nipuntalukdar/kafka_consumer_exporter exports number of topics in  error, also number of partitions in error and in good state,  while https://github.com/danielqsj/kafka_exporter doesn't. Also, https://github.com/nipuntalukdar/kafka_consumer_exporter yet to have a dockerfile added, while https://github.com/danielqsj/kafka_exporter already has a dockerfile. 

On Monday, 13 November 2017 17:53:08 UTC+5:30, Matthias Rampke wrote:

Hi,

This looks really interesting, I'm going to try it out! Can you explain the difference to https://github.com/danielqsj/kafka_exporter ? When would I use one or the other?

Thank you!

/MR

On Sun, Nov 12, 2017 at 6:04 AM Nipun Talukdar <nipun.t...@gmail.com> wrote:

Hi All,

I have added a Kafka consumer lag, offset and topic details monitor for Prometheus.

Please check it here https://github.com/nipuntalukdar/kafka_consumer_exporter

Thanks,

Nipun Talukdar

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

To post to this group, send email to prometheus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/ab688bc9-bcbf-493d-8095-3cb0bde846d6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

To post to this group, send email to prometheus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/e3a07db2-be33-48f8-84f1-3463174438e5%40googlegroups.com.

[Martin Schröder]

2017-11-14 4:00 GMT+01:00 Nipun Talukdar <nipun.t...@gmail.com>:
> I glanced through the source of the other kafka exporter and it really makes
> too many calls to Kafka for getting the stats. For example, to get the

Thanks! Please put that rational in the readme. :-)

Best
Martin

[Nipun Talukdar]

I will update the Readme and also add the metrics for number of in sync replicas and docker file shortly.