Streams Monitoring using Control Center - having issues

[karan alang]

Hello All - i'm having issues in trying to setup Streaming monitoring using Control Center.

what i've done so far :

I have a Java Producer, which produces Avro messages to Confluent Kafka Topic.

To enable streams monitoring, i've done the following

1) added dependency in maven

<dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>${kafka.version}</version>
        </dependency>
        <dependency>
        <groupId>io.confluent</groupId>
        <artifactId>monitoring-interceptors</artifactId>
        <version>${confluent.version}</version>
    </dependency>

2) added following code in Producer Code

props.put("interceptor.classes","io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor");

Then i complied the Producer jar.

I'm planning to do the same with the Java Consumer Code (though not done yet)

On the Broker side ->

I've added the monitoring-interceptors-3.2.2.jar in KAFKA_OPTS -

export KAFKA_OPTS="-Djavax.net.debug=all -javaagent:/usr/hdp/2.5.3.0-37/confluent-3.2.2/share/java/monitoring-interceptors/monitoring-interceptors-3.2.2.jar"

when i start the Broker, it gives me the following error ->

 $CONFLUENT/bin/kafka-server-start $CONFLUENT/mprops/kafka-server.properties
Failed to find Premain-Class manifest attribute in /usr/hdp/2.5.3.0-37/confluent-3.2.2/share/java/monitoring-interceptors/monitoring-interceptors-3.2.2.jar
Error occurred during initialization of VM
agent library failed to init: instrument

The jar i've added to the KAFKA_OPTS is the one that comes bundled with Confluent 3.2.2  

So, couple of questions ->

- What needs to be done to fix this issue ?

- What additional steps to be done to enable Streams monitoring using Control Center  

[karan alang]

additionally (i forgot to add), 

I've enabled Confluent Metrics Reporter in the broker properties file.

confluent.metrics.reporter.bootstrap.servers=<hostname>:9092
confluent.metrics.reporter.zookeeper.connect=<hostname>:3181

I guess, the primary question is - How do i add the monitoring interceptor jar to Kafka Broker Classpath ?

Is setting KAFKA_OPTS the right way or is there another command ?

pls let me know.

thanks!

 

 

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/vRaEsI6h3IY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/78a91ff6-df5f-46ca-bbec-d53198c38cdf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[karan alang]

Hello - here is the update on this.

I believe the classpath issue is resolved .. (just added the jar to the CLASSPATH), however the Stream Monitoring on Confluent Center is still not happenning.

Here is the list of changes done ->

1) added dependency in maven

<dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>${kafka.version}</version>
        </dependency>
        <dependency>
         <groupId>io.confluent</groupId>
         <artifactId>monitoring-interceptors</artifactId>
         <version>${confluent.version}</version>
     </dependency>

2) added following code in Producer Code (which publishes avro messages into Kafka topic)

  props.put("bootstrap.servers", "nwk2-bdp-kafka-04.gdcs-qa.apple.com:9092,nwk2-bdp-kafka-04.gdcs-qa.apple.com:9082,nwk2-bdp-kafka-04.gdcs-qa.apple.com:9072,nwk2-bdp-kafka-04.gdcs-qa.apple.com:9062,nwk2-bdp-kafka-04.gdcs-qa.apple.com:9052");
props.put("acks", "all");
        props.put("retries", 0);
        props.put("key.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer");
        props.put("value.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer");
        props.put("schema.registry.url", schemaUrl);
        //added monitoring interceptor - reference http://docs.confluent.io/current/control-center/docs/clients.html#controlcenter-clients
props.put("producer.interceptor.classes","io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor");

Then i complied the Producer jar.

I'm planning to do the same with the Java Consumer Code (though not done yet)

On the Broker side ->

I've added the monitoring interceptor jar in the classpath,

Also,  

I've enabled Confluent Metrics Reporter in the broker properties file.

confluent.metrics.reporter.bootstrap.servers=<hostname>:9092
confluent.metrics.reporter.zookeeper.connect=<hostname>:3181

Anything else that needs to be done to enable Stream Monitoring using Control Center ?

 

To unsubscribe from this group and all its topics, send an email to confluent-platform+unsubscribe@googlegroups.com.

[karan alang]

fyi, below mentioned shows the monitoring interceptor jars in the kafka broker classpath 

 

ps -ef | grep kafka-server
root     15447  2394  3 00:13 pts/1    00:02:38 /usr/java/jdk1.8.0_121/bin/java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC -Djava.awt.headless=true -Xloggc:/usr/hdp/2.5.3.0-37/confluent-3.2.2/bin/../logs/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/usr/hdp/2.5.3.0-37/confluent-3.2.2/bin/../logs -Dlog4j.configuration=file:/usr/hdp/2.5.3.0-37/confluent-3.2.2/bin/../etc/kafka/log4j.properties -cp :/usr/hdp/2.5.3.0-37/confluent-3.2.2:/usr/hdp/2.5.3.0-37/confluent-3.2.2/share/java/monitoring-interceptors/monitoring-interceptors-3.2.2.jar:/usr/hdp/2.5.3.0-37/confluent-3.2.2/bin/../share/java/kafka/*:/usr/hdp/2.5.3.0-37/confluent-3.2.2/bin/../share/java/confluent-support-metrics/*:/usr/share/java/confluent-support-metrics/* io.confluent.support.metrics.SupportedKafka /usr/hdp/2.5.3.0-37/confluent-3.2.2/mprops/kafka-server.properties

[Gwen Shapira]

Karan,

I think there is some confusion. The interceptors are client-side only. You don't need to configure them on brokers at all.

Gwen

To unsubscribe from this group and all its topics, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/78a91ff6-df5f-46ca-bbec-d53198c38cdf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CABfvUbtKyeY3MY6iD2M5XX4YEaFXummMgC-NYpAKRSP5RWdzxQ%40mail.gmail.com.

[karan alang]

Hi Gwen,

Thanks, will check this out (though i think i saw this mentioned as required while setting up the data Streams Monitoring for the replicator) ..

However, additional question - 

I've noticed that the Streams monitoring shows the streams monitoring only when the Consumer is started. i.e. 

If i'm running the replicator, (from topic - a to topic a.replica), only when i start the consumer to read from the topic a.replica - does the Control Center show the Data Streams Monitoring.

is that a requirement for Streams Monitoring ?

Here is my setup -> 

1) added the following in -> etc/kafka/connect-distributed.properties
# Interceptor setup
consumer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor
producer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor

 

2) Connect classpath has -> monitoring-interceptors-3.2.2.jar

 

3) Replicator setting :
{
        "name":"replicator2",
        "config":{
                "connector.class":"io.confluent.connect.replicator.ReplicatorSourceConnector",
                "tasks.max":4,
                "key.converter":"io.confluent.connect.replicator.util.ByteArrayConverter",
                "value.converter":"io.confluent.connect.replicator.util.ByteArrayConverter",
                "src.kafka.bootstrap.servers":"localhost:8082",
                "src.zookeeper.connect":"localhost:4181",
                "dest.zookeeper.connect":"localhost:3181",
                "topic.whitelist":"repStr2",
                "topic.rename.format":"${topic}.replica",
                "src.consumer.interceptor.classes":"io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor",
                "confluent.license":"<lic key>"
        "src.consumer.group.id":"repl-group"
        }
}

   

 

 

To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/78a91ff6-df5f-46ca-bbec-d53198c38cdf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CABfvUbtKyeY3MY6iD2M5XX4YEaFXummMgC-NYpAKRSP5RWdzxQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/vRaEsI6h3IY/unsubscribe.

To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CAA9QSJ%2BE7k9c4QP6YihQ%3DqfHHSOCGmJHCFoFHkwHm_CWxfrFTA%40mail.gmail.com.

[karan alang]

btw, yes - i was able to setup Streams monitoring (for both replicator & streams within a Kafka cluster) ..

​

[Gwen Shapira]

Yay :)

Streams monitoring compares "actual consumption" to "expected consumption". If the Replicator isn't running at all, we don't expect any consumption, and therefore nothing to show. 

I agree this isn't super intuitive, and we are trying to figure out a better way to show information.

In case of Replicator, I typically set 3 alerts:

* Consumption diff (make sure you don't miss events)

* Consumption lag (are you falling behind?)

* bytes/in on target topics. If too low, Replicator is probably down.

Gwen

On Tue, Aug 22, 2017 at 1:01 PM karan alang <karan...@gmail.com> wrote:

btw, yes - i was able to setup Streams monitoring (for both replicator & streams within a Kafka cluster) ..

​

--

You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CABfvUbvhFZYamQij_2F9Lh083im%3DBAZSmqj0SEvqtQxKeaxPOA%40mail.gmail.com.

[karan alang]

Hi Gwen,

Thanks.. have another question regd. confluent replicator monitoring..

I'm trying to use JMX_PORT to monitor Replicator Producers/Consumer

(ref - http://docs.confluent.io/current/multi-dc/replicator-tuning.html)

Here is what i've done -

1) set JMX_PORT(=12345) & re-started Kafka Connect

2)

Then, i've configured Prometheus yaml to add the Connect  

- job_name: 'connect'
    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s
    static_configs:
      - targets: ['<host where connect runs>:12345']
 

 

3) started the replicator & the producer/consumer

Producer ->

$CONFLUENT1/bin/kafka-producer-perf-test --topic repStr3 --num-records 10000000 --record-size 1000 --throughput 10000 --producer-props bootstrap.servers=hostname:8092 interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor acks=all

Consumer ->

$CONFLUENT/bin/kafka-console-consumer --topic repStr3.replica --new-consumer --bootstrap-server hostname:9092 --consumer.config $CONFLUENT/consumer3.config

However, i don't see the confluent producer/consumer metrics in Prometheus.

so, do i need to do any additional steps to enable monitoring of the Replicator producer/consumer ?

Pls .let me know.

Pls note -> i intent displaying the metrics on Grafana dashboard (by linking Prometheus + Grafana)

thanks! 

On Tue, Aug 22, 2017 at 1:20 PM, Gwen Shapira <gw...@confluent.io> wrote:

Yay :)

Streams monitoring compares "actual consumption" to "expected consumption". If the Replicator isn't running at all, we don't expect any consumption, and therefore nothing to show. 

I agree this isn't super intuitive, and we are trying to figure out a better way to show information.

In case of Replicator, I typically set 3 alerts:

* Consumption diff (make sure you don't miss events)

* Consumption lag (are you falling behind?)

* bytes/in on target topics. If too low, Replicator is probably down.

Gwen

On Tue, Aug 22, 2017 at 1:01 PM karan alang <karan...@gmail.com> wrote:

btw, yes - i was able to setup Streams monitoring (for both replicator & streams within a Kafka cluster) ..

​

--

You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CABfvUbvhFZYamQij_2F9Lh083im%3DBAZSmqj0SEvqtQxKeaxPOA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/vRaEsI6h3IY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CAA9QSJJGurf%2BrgU5x%2BMKhXHAhp7zQnV6HO%2BAQb9GiMvK0-%3DWHA%40mail.gmail.com.

[karan alang]

Or is there a better way to do this ?

possibly view the metrics on the Control Center UI itself (instead of Prometheus - Grafana) ?

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CABfvUbvhFZYamQij_2F9Lh083im%3DBAZSmqj0SEvqtQxKeaxPOA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/vRaEsI6h3IY/unsubscribe.

To unsubscribe from this group and all its topics, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.