Trying to find out how much is buffered up in a Kafka gateway.

[Jayadev Jayaraman]

I have a 2 machine Sensei-1.5.1 cluster [ with 4 partitions per machine ] running that consumes from a Kafka 0.7.0 gateway. Both Sensei and Kafka use the  same 3-machine Zookeeper 3.4.0 ensemble if that is relevant. I pushed a lot of updates to Kafka yesterday and all of them went through the Kafka producer successfully. 

But when I ran queries on my Sensei cluster for certain metrics, some numbers were way off and didn't make sense. I'm suspecting that the Sensei kafka consumer is lagging behind the producer(s) but I don't how to verify this. 

On the Sensei client webapp, I see the number of documents increasing constantly so that means the consumer is working, but I don't know if it's consuming yesterday's stuff that's been buffered up or if it's consuming stuff that was inserted today ( the producers run continuously ) . 

Is there a way I can figure out the kafka buffer size or the number of messages in the buffer just like I can figure out the number of pending messages in a message queue ? Also, is there a way I can view in a log-file in real-time what is being updated in the Sensei gateway ? 

Thanks.

Jayadev

[Roman Garcia]

I believe for Kafka 0.7 there's a JMX operation that lets you analyze the "lag" for a topic/partition.

Just check the Kafka docs, it should be there.

--
You received this message because you are subscribed to the Google Groups "Sensei" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sensei-searc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.