Stream monitoring empty on Confluent Control center

1,180 views
Skip to first unread message

Nabeel Moidu

unread,
Aug 4, 2016, 8:57:53 AM8/4/16
to Confluent Platform
Hi

I've installed a 2 node Kafka cluster on two physical nodes using the confluent platform rpms and have faced the classpath issues Vadim reported earlier. I've set an explicit classpath and worked around it. Now the UI comes up, and I can create sample file sink and source for a particular topic (used the nohup.out from the console log messages so that there's always content in it). The file is replicated at the sink location and at the kafka layer things seem fine.

I was expecting to see some activity graphs in the Stream monitoring section on the Control Center UI as per the demo video on the website. But there's no activity shown on the UI. I tried changing the time slots but it's had no effect. I'm not sure what log would give any pointers. The fd pointers in the proc dir for the control center don't lead to any interesting info.

Nabeel

Nabeel Moidu

unread,
Aug 4, 2016, 9:10:53 AM8/4/16
to Confluent Platform
Looking at the browser JS console, it seems to throw 404 errors similar to what was reported by Yogesh Sangvikar earlier last month. I already have the config below in my properties file

zookeeper.connect=XXXX.XXX:2181
confluent.controlcenter.internal.topics.partitions=1
confluent.controlcenter.internal.topics.replication=1
confluent.monitoring.interceptor.topic.partitions=1
confluent.monitoring.interceptor.topic.replication=1

It doesn't resolve the issue though.

Nabeel

Roger Hoover

unread,
Aug 4, 2016, 1:50:39 PM8/4/16
to confluent...@googlegroups.com
Hi Nabeel,

My guess is that monitoring data is not being generated.  Perhaps the interceptors were not configured for Connect?  Can you double check that these properties are configured for connect?

consumer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor
producer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor

Can you run kafka-console-consumer on the _confluent-monitoring topic and see if any data is in it? 

Cheers,

Roger

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/b0e45001-5de0-4dd0-bd69-9e9eef214fc7%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Nabeel Hyd

unread,
Aug 4, 2016, 9:37:29 PM8/4/16
to confluent...@googlegroups.com
[root@XXXXXX ~]# kafka-console-consumer --zookeeper=XXXXX.XXX:2181 --topic=_confluent-monitoring --from-beginning
===SNIP===
^CProcessed a total of 12319 messages

[root@XXXXXX ~]# cat /tmp/connect-distributed.properties  | grep -v ^# | grep -v ^$
consumer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor
producer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor
bootstrap.servers=XXXXX:9092
group.id=connect-cluster
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=true
value.converter.schemas.enable=true
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
offset.storage.topic=connect-offsets
config.storage.topic=connect-configs
status.storage.topic=connect-status

Looks fine to me.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/b0e45001-5de0-4dd0-bd69-9e9eef214fc7%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/AaMBOCijXcE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CAPOm%3DTPGr7vLm_vqXWVKQgbNZ5WA3uNGYfR7mUc3EP-MRYqBeA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
Thanks and Regards,

Nabeel Moidu
Hyderabad, India

Roger Hoover

unread,
Aug 8, 2016, 1:39:39 PM8/8/16
to confluent...@googlegroups.com
Hi Nabeel,

There are a few scenarios I can think of that might explain why you don't see monitored data.

1.  Time issues - the data will be presented in the Control Center UI based on the timestamp that the producer assigned to each message.  If the date were off significantly on the machine where the producers were running, it could cause the data not to show up in the expected time range in the UI.
2.  No monitoring data has arrived.  Although you found 12k messages in the _confluent-monitoring topic, it's possible that those are only heartbeat messages that Control Center sends to itself.
3.  Only producer monitoring data arrived.  If no matching consumer data arrived, the UI does not display audit data (since it's goal is to match consumption with production).
4.  There a fatal error in the Kafka Stream application inside Control Center.  Is there anything in the Control Center log or perhaps a stack dump will help here.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/AaMBOCijXcE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CAPOm%3DTPGr7vLm_vqXWVKQgbNZ5WA3uNGYfR7mUc3EP-MRYqBeA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
Thanks and Regards,

Nabeel Moidu
Hyderabad, India

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CAKTAHSzJjpKqBPwGmwPEUsPT99eY9N5g3V9znzdQQwiH7gu4sw%40mail.gmail.com.

Alexander Jipa

unread,
Aug 9, 2016, 2:16:41 PM8/9/16
to Confluent Platform
Hello,
I'm having the same issue - everything is up and running, control center shows the source task but no messages metrics, not even the topics.
The _confluent-monitoring topic is empty, the only exception in connect log:
[2016-08-09 14:13:14,686] ERROR Unhandled exception when committing WorkerSourceTask{id=Orders-0}:  (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitt
er:115)
org.apache.kafka.connect.errors.ConnectException: OffsetStorageWriter is already flushing
        at org.apache.kafka.connect.storage.OffsetStorageWriter.beginFlush(OffsetStorageWriter.java:112)
        at org.apache.kafka.connect.runtime.WorkerSourceTask.commitOffsets(WorkerSourceTask.java:279)
        at org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter.commit(SourceTaskOffsetCommitter.java:107)
        at org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter.access$000(SourceTaskOffsetCommitter.java:44)
        at org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter$1.run(SourceTaskOffsetCommitter.java:73)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
How can I make sure the metrics are published?

Could you please elaborate on point 3.?
Does this mean that even though my source task is configured for a Kafka Connect that has interceptors enabled - we won't see the statistics unless there's a consumer?

---
Cheers,
Alexander

To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/AaMBOCijXcE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.



--
Thanks and Regards,

Nabeel Moidu
Hyderabad, India

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

Roger Hoover

unread,
Aug 9, 2016, 2:43:48 PM8/9/16
to confluent...@googlegroups.com
Hi Alexander,

>> How can I make sure the metrics are published?

The monitoring interceptor needs to be configured and on the classpath (http://docs.confluent.io/3.0.0/control-center/docs/clients.html).  There's an INFO level log I think you can use to verify that the interceptor is working.  It starts with "creating producer for client=".

>> Could you please elaborate on point 3.? Does this mean that even though my source task is configured for a Kafka Connect that has interceptors enabled - we won't see the statistics unless there's a consumer?

Yes, the current UI is comparing production versus "expected consumption" for each topic partition.  I say "expected consumption" because if you have 2 consumer groups configured with the monitoring interceptor, then we expect all messages to be consumed twice.  In the case where you have a source task producing data to Kafka and no consumers to read it, the UI does not have any "expected consumption" yet.

Here's a way you could consume from the console consumer.

echo "interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor" > /tmp/consumer.config
/usr/bin/control-center-run-class kafka.tools.ConsoleConsumer --bootstrap-server "$BOOTSTRAP_SERVERS" --topic "$TOPIC" --new-consumer --from-beginning --consumer.config /tmp/consumer.config

Roger

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/AaMBOCijXcE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.



--
Thanks and Regards,

Nabeel Moidu
Hyderabad, India

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

Alexander Jipa

unread,
Aug 9, 2016, 3:20:36 PM8/9/16
to Confluent Platform
Hi Roger,
Thanks for the clarification. Is this logic mentioned anywhere in the Control Center documentation?

In my case the producer interceptor is created - no issues on this side it seems.

I've tried running the console consumer as suggested.
It takes a while to start and even manages to print a message (I've noticed that this is usually the last message for the topic).
But Control Center is still not reflecting anything.
More over that I cannot find the offset for this consumer stored anywhere and the consumer itself is stuck awaiting for new data...

Restarting Kafka Connect to replay the feed - do not seem to help too.
Consumer shows new data, and then heartbeat:
[2016-08-09 15:18:49,059] DEBUG Publish clientId: "consumer-1"
group: "console-consumer-37155"
session: "e718df46-a3a9-428c-8eca-b92f9537b3e8"
sequence: 164
timestamp: 1470770329058
topic: "test-Topic"
samplePeriod: 15000
type: HEARTBEAT
minWindow: 1470770235000
maxWindow: 1470770265000
 (io.confluent.monitoring.clients.interceptor.MonitoringInterceptor:250)

:(

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/AaMBOCijXcE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.



--
Thanks and Regards,

Nabeel Moidu
Hyderabad, India

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

Alexander Jipa

unread,
Aug 9, 2016, 3:27:44 PM8/9/16
to Confluent Platform
Hi,
In addition I've noticed the following in Control Center:
[2016-08-09 15:20:47,671] DEBUG RESPONSE /1.0/monitoring/consumer_groups  404 handled=true (org.eclipse.jetty.server.Server:502)
[2016-08-09 15:17:19,288] DEBUG RESPONSE /1.0/monitoring/consumer_groups_topics  404 handled=true (org.eclipse.jetty.server.Server:502)
And _confluent.monitoring is still empty...

---
Best Regards,
Alexander

Roger Hoover

unread,
Aug 9, 2016, 4:10:14 PM8/9/16
to confluent...@googlegroups.com
Alexander,

There's some explanation here.  Let us know if it can be more clear.


"The delivery chart shows the number of expected messages as a line on top, and the number consumed as the area. (The number of expected messages is the number of messages produced on topics for which there are consumers.) A gap between the “produced” lines and the “consumed” area indicates that some messages that were produced have not yet been seen by the consumer."

Is there data in "test-Topic"?  You said the console consumer showed new data?  Did it send a non-heartbeat message?  What does the timestamp is in the non-heartbeat message?

_confluent-monitoring topic should definitely not be empty, especially because you're seeing the debug log below where the consumer interceptor is publishing a heartbeat message.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/AaMBOCijXcE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.



--
Thanks and Regards,

Nabeel Moidu
Hyderabad, India

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

Alexander Jipa

unread,
Aug 9, 2016, 4:23:08 PM8/9/16
to Confluent Platform
Hello,
Thanks for the link to the detailed description. It cannot be more clear, I simply missed that section.
The data is present in test-Topic, any console-consumer that I create manages to get the data.

Example of a non-heartbeat message that I've seen:
group: "console-consumer-37155"
session: "e718df46-a3a9-428c-8eca-b92f9537b3e8"
sequence: 406
window: 1470773940000
timestamp: 1470773959300
topic: "test-Topic"
count: 2
aggregateBytes: 1697
aggregateCrc: -633947490
totalLatency: 3
minLatency: 1
maxLatency: 2
samplePeriod: 15000
minWindow: -1
maxWindow: -1
 (io.confluent.monitoring.clients.interceptor.MonitoringInterceptor:250)

_confluent.monitoring has no data for sure:
./bin/kafka-console-consumer --zookeeper <ZK> --topic _confluent.monitoring --from-beginning
^CProcessed a total of 0 messages

I've mentioned that I cannot see the offsets for this Control-Center's consumer.
Can you please help me debug/troubleshoot this further?

Thank you!

---
Cheers,
Alexander
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/AaMBOCijXcE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.



--
Thanks and Regards,

Nabeel Moidu
Hyderabad, India

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

Alexander Jipa

unread,
Aug 9, 2016, 4:27:04 PM8/9/16
to Confluent Platform
Hello,
Also wanted to mention that I use KafkaOffsetMonitor (https://github.com/quantifind/KafkaOffsetMonitor) to see the active consumers for all topics in my Kafka cluster.
And the only active consumer for _confluent.monitoring is my console consumer.
Shouldn't Control-Center actively monitor this topic?

---
Cheers,
Alexander

Roger Hoover

unread,
Aug 9, 2016, 6:16:30 PM8/9/16
to confluent...@googlegroups.com
Ah, ok.  I think the reason you're not seeing the offsets in KafkaOffsetMonitor is because it only works with the old consumer which stores it's offsets in ZooKeeper.  I've run burrow (https://github.com/linkedin/Burrow) to see the offsets or you can run this cmd line tool.  

/bin/kafka-run-class.sh kafka.admin.ConsumerGroupCommand --bootstrap-server localhost:9092 --new-consumer --list
./bin/kafka-run-class.sh kafka.admin.ConsumerGroupCommand --bootstrap-server localhost:9092 --new-consumer --describe --group _confluent-controlcenter-0

It's strange that you don't see any data in _confluent-monitoring topic.  There is a config to change the name of that topic in both the interceptors and Control Center (confluent.monitoring.interceptor.topic).  Any chance you set that to something else?

Alexander
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/AaMBOCijXcE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.



--
Thanks and Regards,

Nabeel Moidu
Hyderabad, India

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/0576bd00-935e-4dfd-a224-947958aa311a%40googlegroups.com.

Alexander Jipa

unread,
Aug 9, 2016, 6:34:43 PM8/9/16
to Confluent Platform
Hi Roger,
I'll try Burrow, thanks!

This is how my monitoring topic is configured in control-center config:
confluent.monitoring.interceptor.topic=_confluent-monitoring

I've not configured it for the interceptors...

As for the first command it returns: _confluent-controlcenter-0

As for the second: all the topics have 
CURRENT-OFFSET=unknown  LOG-END-OFFSET=0  LAG=unknown OWNER=_confluent-controlcenter-0-1-StreamThread-1-consumer_/<IP>

---
Cheers,
Alexander
Alexander
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/AaMBOCijXcE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.



--
Thanks and Regards,

Nabeel Moidu
Hyderabad, India

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

Roger Hoover

unread,
Aug 9, 2016, 7:28:20 PM8/9/16
to confluent...@googlegroups.com
Hmmm...strange.  Does the _confluent-monitoring topic exist?  If not, producers and consumers will block until it's created.

Alexander
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/AaMBOCijXcE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.



--
Thanks and Regards,

Nabeel Moidu
Hyderabad, India

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/a8e57faa-a804-4dd8-aeb2-97a65c826cb6%40googlegroups.com.

Nabeel Hyd

unread,
Aug 10, 2016, 5:38:21 AM8/10/16
to confluent...@googlegroups.com
On Tue, Aug 9, 2016 at 1:39 AM, Roger Hoover <roger....@gmail.com> wrote:
Hi Nabeel,


Thanks for the detailed response.
 
There are a few scenarios I can think of that might explain why you don't see monitored data.

1.  Time issues - the data will be presented in the Control Center UI based on the timestamp that the producer assigned to each message.  If the date were off significantly on the machine where the producers were running, it could cause the data not to show up in the expected time range in the UI.

The producer and UI are on the same node. I don't believe this may be a factor here.
 
2.  No monitoring data has arrived.  Although you found 12k messages in the _confluent-monitoring topic, it's possible that those are only heartbeat messages that Control Center sends to itself.
How can I distinguish between the two ? I can now see 100K+ messages in the consumer console for the topic.
 
3.  Only producer monitoring data arrived.  If no matching consumer data arrived, the UI does not display audit data (since it's goal is to match consumption with production).
I've got a source and a sink, and the two files are completely in sync, so I assume both producer and consumer are working fine. Not sure about the monitoring data though. Is there a way I can check them individually ?

 4.  There a fatal error in the Kafka Stream application inside Control Center.  Is there anything in the Control Center log or perhaps a stack dump will help here.

I've pasted the jstack dumps at :

Doesn't look like anything's blocked there.

Nabeel


To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/AaMBOCijXcE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.



--
Thanks and Regards,

Nabeel Moidu
Hyderabad, India

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/AaMBOCijXcE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Alexander Jipa

unread,
Aug 10, 2016, 10:02:04 AM8/10/16
to Confluent Platform
Hello,
The topic exists and I'm able to consume it via the console - no error, just black screen.
Is there a way of debugging this?
Did non-heartbeat message indicate anything?

---
Cheers,
Alexander
Alexander
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/AaMBOCijXcE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.



--
Thanks and Regards,

Nabeel Moidu
Hyderabad, India

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

Roger Hoover

unread,
Aug 10, 2016, 3:10:08 PM8/10/16
to confluent...@googlegroups.com
Nabeel,

Based on your stack dumps, it looks to me like Control Center did not initialize successfully.  When it's fully up and running, there should be threads with names like "StreamThread-1".  Were there errors in the log?  Either way, I think it's work increasing the log verbosity.

A few more comments below.

Cheers,

Roger


On Wed, Aug 10, 2016 at 2:37 AM, Nabeel Hyd <nabee...@gmail.com> wrote:


On Tue, Aug 9, 2016 at 1:39 AM, Roger Hoover <roger....@gmail.com> wrote:
Hi Nabeel,


Thanks for the detailed response.
 
There are a few scenarios I can think of that might explain why you don't see monitored data.

1.  Time issues - the data will be presented in the Control Center UI based on the timestamp that the producer assigned to each message.  If the date were off significantly on the machine where the producers were running, it could cause the data not to show up in the expected time range in the UI.

The producer and UI are on the same node. I don't believe this may be a factor here.
 
2.  No monitoring data has arrived.  Although you found 12k messages in the _confluent-monitoring topic, it's possible that those are only heartbeat messages that Control Center sends to itself.
How can I distinguish between the two ? I can now see 100K+ messages in the consumer console for the topic.

Even though the data is binary, you should see strings in there for consumer interceptor messages with the consumer group name.   You could also turn the logging to DEBUG or TRACE in control center and there should be a lot more clues.
 
 
3.  Only producer monitoring data arrived.  If no matching consumer data arrived, the UI does not display audit data (since it's goal is to match consumption with production).
I've got a source and a sink, and the two files are completely in sync, so I assume both producer and consumer are working fine. Not sure about the monitoring data though. Is there a way I can check them individually ?

Same as above.  The data should have human readable consumer group and topic strings.  Turning up the logging verbosity in Control Center should give more clues.
 
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.



--
Thanks and Regards,

Nabeel Moidu
Hyderabad, India

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.

Roger Hoover

unread,
Aug 10, 2016, 3:16:22 PM8/10/16
to confluent...@googlegroups.com
The non-heartbeat message suggested that the monitoring interceptor data is being sent to the _confluent-monitoring topic but you aren't able to see it with the console consumer.  It seems like something is wrong with either Kafka or ZooKeeper in this environment.

One thing you could do is to look at the Kafka data directory and see if the .log files for the _confluent-monitoring topic are > 0 bytes.

Alexander
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/AaMBOCijXcE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.



--
Thanks and Regards,

Nabeel Moidu
Hyderabad, India

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/d7f3b9fc-c96b-4260-9b4b-54d95f624266%40googlegroups.com.

Nabeel Hyd

unread,
Aug 11, 2016, 3:58:27 AM8/11/16
to confluent...@googlegroups.com
I've started the control center process with DEBUG logging enabled. There's one notable error I can see :
[2016-08-11 07:29:59,154] ERROR streams exception (io.confluent.controlcenter.streams.KafkaStreamsManager:36)
java.lang.IllegalAccessError: tried to access class com.google.common.collect.TreeRangeSet from class io.confluent.controlcenter.streams.verify.MonitoringVerifier
    at io.confluent.controlcenter.streams.verify.MonitoringVerifier.<init>(MonitoringVerifier.java:61)
    at io.confluent.controlcenter.streams.verify.MonitoringVerifier.<init>(MonitoringVerifier.java:84)
    at io.confluent.controlcenter.streams.verify.VerifyTransformerSupplier$MonitoringVerifyTransformer.transform(VerifyTransformerSupplier.java:170)
    at io.confluent.controlcenter.streams.verify.VerifyTransformerSupplier$MonitoringVerifyTransformer.transform(VerifyTransformerSupplier.java:114)
    at org.apache.kafka.streams.kstream.internals.KStreamTransform$KStreamTransformProcessor.process(KStreamTransform.java:57)
    at org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:68)
    at org.apache.kafka.streams.processor.internals.StreamTask.forward(StreamTask.java:338)
    at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:187)
    at org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:64)
    at org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:174)
    at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:320)
    at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:218)
[2016-08-11 07:29:59,157] DEBUG Stopping Kafka Stream process (org.apache.kafka.streams.KafkaStreams:186)
[2016-08-11 07:29:59,157] DEBUG Stopping Kafka Stream process (org.apache.kafka.streams.KafkaStreams:186)
[2016-08-11 07:29:59,242] DEBUG Shutting down at user request. (org.apache.kafka.streams.processor.internals.StreamThread:394)
[2016-08-11 07:29:59,242] DEBUG Shutting down at user request. (org.apache.kafka.streams.processor.internals.StreamThread:394)

The full log can be seen here . http://pastebin.com/jSS4VPgV

I still don't see the StreamThread in the jstack dump, so I think the above error maybe linked to the empty graphs. I do see ASCII strings in the console consumer output related to the sample topic (nohup) :

[ user@kafkanode ~]# kafka-console-consumer --zookeeper=XXXX.XX:2181 --topic=_confluent-monitoring 

producer-1"$d33a1a2a-7b3a-41f1-b8b0-6bb0f0af16e1(�� 0�����*8�����*BnohupH P
                                                                           X� `����� �u� �����*� �����*
producer-1"$d33a1a2a-7b3a-41f1-b8b0-6bb0f0af16e1(�� 0�����*8�����*BnohupH P X� `����� �u� �����*� �����*
consumer-4 connect-nohupsink"$4e0514c7-3c8c-47ad-a550-a94594811eb6(�� 0�����*8�����*BnohupH P
                                                                                              X� `����h x � �u� �����*� �����*
consumer-4 connect-nohupsink"$4e0514c7-3c8c-47ad-a550-a94594811eb6(�� 0�����*8�����*BnohupH P X� `����h p x � �u� �����*� �����*
^CProcessed a total of 5 messages




--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/AaMBOCijXcE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CAPOm%3DTOo2bSTMt3Zsn_RC6_-UGd0OxtY54ka2SG1NukxqLt%3DQA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

Roger Hoover

unread,
Aug 11, 2016, 8:01:31 PM8/11/16
to confluent...@googlegroups.com
Hi Nabeel,

That explains why nothing is working but I don't know how it would get an IllegalAccessError instantiating that class.


To unsubscribe from this group and all its topics, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CAPOm%3DTOo2bSTMt3Zsn_RC6_-UGd0OxtY54ka2SG1NukxqLt%3DQA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
Thanks and Regards,

Nabeel Moidu
Hyderabad, India

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CAKTAHSwdspsw%2BfjFa4GFntWs2WW0TPyJrH7PCwvzFTTBYQQjTA%40mail.gmail.com.

Nabeel Hyd

unread,
Aug 12, 2016, 5:28:34 AM8/12/16
to confluent...@googlegroups.com
Thanks Roger.

What guava versions are known to work for the control center ?

Nabeel

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CAKTAHSwdspsw%2BfjFa4GFntWs2WW0TPyJrH7PCwvzFTTBYQQjTA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/AaMBOCijXcE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CAPOm%3DTPV_u5O52bwsDBjxqwSQG1%2B9wFGW3mQGk1zeKYRAqnvGQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

Nabeel Hyd

unread,
Aug 12, 2016, 5:42:54 AM8/12/16
to confluent...@googlegroups.com
Ok, managed to fix this. When I removed the guava-13.0-6.el7.noarch guava jar from the CLASSPATH, it would throw an error :

[2016-08-12 09:31:28,124] ERROR streams exception (io.confluent.controlcenter.streams.KafkaStreamsManager:36)
java.lang.NoSuchMethodError: com.google.common.collect.RangeSet.asDescendingSetOfRanges()Ljava/util/Set;
        at io.confluent.controlcenter.streams.verify.MonitoringVerifier.getMissingMonitoringData(MonitoringVerifier.java:304)

Then looking for all guava jars, turned out the confluent platform itself shipped multiple guava jars, and the older one was causing the issue above

[ro...@XXx.xx ~]# for i in `rpm -qa | grep confluent`; do rpm -ql $i | grep jar$ | grep guava; done
/usr/share/java/schema-registry/jersey-guava-2.19.jar
/usr/share/java/kafka/guava-18.0.jar
/usr/share/java/kafka/jersey-guava-2.22.2.jar
/usr/share/java/confluent-control-center/guava-19.0.jar
/usr/share/java/confluent-control-center/jersey-guava-2.19.jar
/usr/share/java/monitoring-interceptors/guava-19.0.jar
/usr/share/java/kafka-connect-hdfs/guava-14.0.1.jar
/usr/share/java/rest-utils/jersey-guava-2.19.jar

I removed guava-18.0.jar from that location, and now the control center is working. The graphs are finally showing up.




To unsubscribe from this group and all its topics, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.



--
Thanks and Regards,

Nabeel Moidu
Hyderabad, India

Roger Hoover

unread,
Aug 12, 2016, 3:02:26 PM8/12/16
to confluent...@googlegroups.com
Hi Nabeel,

Thanks for tracking down the issue and letting us know.  Looks like you already has the CLASSPATH environment variable defined with /usr/share/java/kafka on it and there's a version conflict between Kafka and Control Center.  I've made a change that will come in the next release of Confluent Platform to minimize the chance of these conflicts by putting Control Center's dependencies ahead of whatever gets passed in via environment variable.

Cheers,

Roger

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CAKTAHSwhxWLUOZwbp0DNfsWXrETAgWM--VakZCqN1SFJLMy4Tg%40mail.gmail.com.

Nabeel Hyd

unread,
Aug 13, 2016, 1:56:40 AM8/13/16
to confluent...@googlegroups.com
Thanks Roger.

Yes I had to explicitly specify it in the order in which my trial and error attempts were failing. 

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CAKTAHSwhxWLUOZwbp0DNfsWXrETAgWM--VakZCqN1SFJLMy4Tg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/AaMBOCijXcE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CAPOm%3DTO2gDWNv%3D0bcayY2S38ZfqvBqBmc07yKawbQt20JA8LWA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages