Control center with client interceptor

[Jun MA]

Hi team,

I'm a little confused about how can we use the control center. I'm aware that it can monitor kafka connect, but I'm not sure if it can monitor metrics that is not from kafka connect. I've followed the guide by adding following config to producer and consumer.

properties.put("interceptor.classes", "io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor");

properties.put("interceptor.classes", "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor");

and run the producer and consumer as usual.

I didn't see anything appear on control center. Also there is nothing new shows up when creating new source. 

Is this the correct way to get client metrics? Or I can only use kafka connect to get metrics?

Thanks,

Ju

[Ewen Cheslack-Postava]

Yes, this is all you should need to do. I assume you are only adding the first line to your producers, and the second to your consumers.

Another requirement is that you are using 0.10.0.0 clients. This was the first version that the interceptors were added. Using clients before 0.10.0 will just result in a log warning about the interceptor.classes setting not being used.

-Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/703bb5e8-7317-4f70-a30e-d92940a7e7cf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Ewen

[Jun MA]

Hi Ewen,

Thank you so much for your reply. I'm using 0.10.0.0-cp1 kafka client and using confluent cp3 kafka brokers. I cannot see anything from Stream Monitoring. Is this specific for kafka stream? I'm using regular consumer and producer, not kafka stream.

I do see some 404 in control center log, not sure if this is related:

"GET /1.0/monitoring/consumer_groups_topics?startTimeMs=1464885000000&stopTimeMs=1464899400000&rollup=FIFTEEN_SECONDS HTTP/1.1" 404 4500  312 (io.confluent.rest-utils.requests:77)

The following is my control center configuration, I'm running everything locally.

bootstrap.servers=localhost:9092

confluent.controlcenter.name=_confluent-controlcenter

confluent.controlcenter.id=0

confluent.controlcenter.data.dir=/tmp/confluent/control-center

confluent.controlcenter.connect.cluster=localhost:8083

zookeeper.connect=localhost:2181

port=9021

confluent.controlcenter.internal.topics.partitions=1

confluent.controlcenter.internal.topics.replication=1

confluent.monitoring.interceptor.topic.partitions=1

confluent.monitoring.interceptor.topic.replication=1

debug=true

On Wednesday, June 1, 2016 at 11:24:09 PM UTC-7, Ewen Cheslack-Postava wrote:

Yes, this is all you should need to do. I assume you are only adding the first line to your producers, and the second to your consumers.

Another requirement is that you are using 0.10.0.0 clients. This was the first version that the interceptors were added. Using clients before 0.10.0 will just result in a log warning about the interceptor.classes setting not being used.

-Ewen

On Wed, Jun 1, 2016 at 7:23 PM, Jun MA <mj.sab...@gmail.com> wrote:

Hi team,

I'm a little confused about how can we use the control center. I'm aware that it can monitor kafka connect, but I'm not sure if it can monitor metrics that is not from kafka connect. I've followed the guide by adding following config to producer and consumer.

properties.put("interceptor.classes", "io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor");

properties.put("interceptor.classes", "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor");

and run the producer and consumer as usual.

I didn't see anything appear on control center. Also there is nothing new shows up when creating new source. 

Is this the correct way to get client metrics? Or I can only use kafka connect to get metrics?

Thanks,

Ju

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/703bb5e8-7317-4f70-a30e-d92940a7e7cf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Ewen

[Jun MA]

Okay I figured that out. I used to use kafka client 0.9.0.1, and after I changed the version in pom file, it actually not changed in the classpath, so it still used the old version.

Thanks for your help!

On Wednesday, June 1, 2016 at 11:24:09 PM UTC-7, Ewen Cheslack-Postava wrote:

Yes, this is all you should need to do. I assume you are only adding the first line to your producers, and the second to your consumers.

Another requirement is that you are using 0.10.0.0 clients. This was the first version that the interceptors were added. Using clients before 0.10.0 will just result in a log warning about the interceptor.classes setting not being used.

-Ewen

On Wed, Jun 1, 2016 at 7:23 PM, Jun MA <mj.sab...@gmail.com> wrote:

Hi team,

I'm a little confused about how can we use the control center. I'm aware that it can monitor kafka connect, but I'm not sure if it can monitor metrics that is not from kafka connect. I've followed the guide by adding following config to producer and consumer.

properties.put("interceptor.classes", "io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor");

properties.put("interceptor.classes", "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor");

and run the producer and consumer as usual.

I didn't see anything appear on control center. Also there is nothing new shows up when creating new source. 

Is this the correct way to get client metrics? Or I can only use kafka connect to get metrics?

Thanks,

Ju

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/703bb5e8-7317-4f70-a30e-d92940a7e7cf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Ewen

[Tianxiang Xiong]

In Confluent Platform 3.0.1, the configuration keys should be `consumer.interceptor.classes` and `producer.interceptor.classes`, right? The documentation still says `interceptor.classes`. http://docs.confluent.io/3.0.1/control-center/docs/clients.html#configuring-applications-to-use-the-confluent-monitoring-interceptor

[Roger Hoover]

Tianxiang,

Thank you for reporting this.  We'll get the documentation fixed.

Cheers,

Roger

On Tue, Oct 18, 2016 at 10:40 AM, Tianxiang Xiong <tianxia...@fundingcircle.com> wrote:

In Confluent Platform 3.0.1, the configuration keys should be `consumer.interceptor.classes` and `producer.interceptor.classes`, right? The documentation still says `interceptor.classes`. http://docs.confluent.io/3.0.1/control-center/docs/clients.html#configuring-applications-to-use-the-confluent-monitoring-interceptor

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/fd7de25b-6d8b-4ee5-bc9b-dc976c9e85ea%40googlegroups.com.

[Tianxiang Xiong]

Good to hear! While you're at it, there are several locations that say the default monitoring topic is `_confluent.monitoring`, while it should be `_confluent-monitoring`.

On Tuesday, 18 October 2016 10:52:08 UTC-7, Roger Hoover wrote:

Tianxiang,

Thank you for reporting this.  We'll get the documentation fixed.

Cheers,

Roger

On Tue, Oct 18, 2016 at 10:40 AM, Tianxiang Xiong <tianxia...@fundingcircle.com> wrote:

In Confluent Platform 3.0.1, the configuration keys should be `consumer.interceptor.classes` and `producer.interceptor.classes`, right? The documentation still says `interceptor.classes`. http://docs.confluent.io/3.0.1/control-center/docs/clients.html#configuring-applications-to-use-the-confluent-monitoring-interceptor

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

[Roger Hoover]

/me slaps forehead.

Thanks, I'll look for those as well.

On Tue, Oct 18, 2016 at 3:47 PM, Tianxiang Xiong <tianxia...@fundingcircle.com> wrote:

Good to hear! While you're at it, there are several locations that say the default monitoring topic is `_confluent.monitoring`, while it should be `_confluent-monitoring`.

On Tuesday, 18 October 2016 10:52:08 UTC-7, Roger Hoover wrote:

Tianxiang,

Thank you for reporting this.  We'll get the documentation fixed.

Cheers,

Roger

On Tue, Oct 18, 2016 at 10:40 AM, Tianxiang Xiong <tianxia...@fundingcircle.com> wrote:

In Confluent Platform 3.0.1, the configuration keys should be `consumer.interceptor.classes` and `producer.interceptor.classes`, right? The documentation still says `interceptor.classes`. http://docs.confluent.io/3.0.1/control-center/docs/clients.html#configuring-applications-to-use-the-confluent-monitoring-interceptor

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/fd7de25b-6d8b-4ee5-bc9b-dc976c9e85ea%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/46c4705d-becc-47c2-b37c-aad92fbc1329%40googlegroups.com.