How to monitor consumer lag in Kafka Indexing Service

[anu...@getblueshift.com]

Hi,

I am using Kafka Indexing Service to ingest data into druid(0.9.2). I would like to monitor the consumer lag metrics for indexing Kafka consumer.

I initially thought I would use kafka.consumer lag metrics but later looked at Kafka zookeeper and realized that Kafka Indexing Service doesn't commit offsets to Kafka. I am assuming that offsets are stored in Druid's metastore table.

Druid on the other end doesn't emit any consumer lag metric(http://druid.io/docs/latest/operations/metrics.html). 

Is there any other way to monitor consumer lag? Another idea which comes to my mind is to pass auto commit properties in "consumerProperties" section of Supervisor spec but not sure, that might lead to some side effects in Kafka ingestion.

Thanks

Anuraj

[Gian Merlino]

Druid does the tracking in its own metadata store, so that's why you don't see them in Kafka. The supervisor status report has the current offsets under "currentOffsets" for each task. Check out "Get Supervisor Status Report" on http://druid.io/docs/latest/development/extensions-core/kafka-ingestion.html.

Gian

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+unsubscribe@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/14a3d904-4338-49a7-8a5d-475b6a24529e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[anu...@getblueshift.com]

Thanks for the info Gian. I think "Get Supervisor Status Report" is good for our current monitoring use case.

On Saturday, December 17, 2016 at 10:40:32 AM UTC-8, Gian Merlino wrote:

Druid does the tracking in its own metadata store, so that's why you don't see them in Kafka. The supervisor status report has the current offsets under "currentOffsets" for each task. Check out "Get Supervisor Status Report" on http://druid.io/docs/latest/development/extensions-core/kafka-ingestion.html.

Gian

On Fri, Dec 16, 2016 at 4:32 PM, <anu...@getblueshift.com> wrote:

Hi,

I am using Kafka Indexing Service to ingest data into druid(0.9.2). I would like to monitor the consumer lag metrics for indexing Kafka consumer.

I initially thought I would use kafka.consumer lag metrics but later looked at Kafka zookeeper and realized that Kafka Indexing Service doesn't commit offsets to Kafka. I am assuming that offsets are stored in Druid's metastore table.

Druid on the other end doesn't emit any consumer lag metric(http://druid.io/docs/latest/operations/metrics.html). 

Is there any other way to monitor consumer lag? Another idea which comes to my mind is to pass auto commit properties in "consumerProperties" section of Supervisor spec but not sure, that might lead to some side effects in Kafka ingestion.

Thanks

Anuraj

--
You received this message because you are subscribed to the Google Groups "Druid User" group.

To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.