Kafka Monitoring: What's the best implementation for Monitoring?

[SMJ]

We're using Kafka for our current data pipeline. We need to implement a monitory application to compare the records produced and the records in the HDFS for each hour. We've thought about a few implementations:

1) The Producer sends an Audit Record about the number of records it has produced in the past one hour. The Consumer application does the same and they're both compared. But there are many loopholes to this. Is there any other way this can be done? 

NOTE: We have a HDFS sink connector for injecting records to HDFS. Can we include monitoring in Connector itself? If so, how?

Please help! 

Thanks in advance

[Robin Moffatt]

One option to look at is Confluent Control Center. Amongst other things, this can track and alert you to disparity between records produced vs consumed.

See https://www.confluent.io/product/control-center/ and for a live demo https://www.confluent.io/online-talk/monitoring-and-alerting-apache-kafka-with-confluent-control-center/

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/401710a0-4024-46b0-b9f6-0b7b8958aa38%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.