Metrics Reporting for Connect

[Eric Pheatt]

Should the kafka.metrics.reporters for the satellite confluent services such as connect, rest-proxy and schema-registry work out of the box or is the KafkaMetricsReporter interface just intended for broker metrics?

I'm evaluating the graphite reporter provided by https://github.com/apakulov/kafka-graphite and it is working well enough for monitoring the broker but it doesn't look like the other confluent services participate directly with that Reporter framework. 

Thanks,

Eric

[Ewen Cheslack-Postava]

That class is only for the broker. org.apache.kafka.common.metrics.MetricsReporter is the one you'd want to use. It will work with everything that ships with AK (i.e. Kafka, Java clients, Connect, Streams). Schema Registry, Rest Proxy and other Confluent products use a different, but essentially the same interface: https://github.com/confluentinc/common/blob/master/metrics/src/main/java/io/confluent/common/metrics/MetricsReporter.java

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/9b532aa4-fe19-4173-ad27-5e3aaae79052%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Eric Pheatt]

For connect it looks like when I set the following in the worker.properties file:

kafka.metrics.reporters=org.apache.kafka.common.metrics.MetricsReporter

kafka.graphite.metrics.reporter.enabled=true

kafka.graphite.metrics.host=graphite.hostname.com

kafka.graphite.metrics.port=2003

kafka.graphite.metrics.prefix=collectd.shared.KAFKAC.shared-kfkc-1

they are not getting reflected by the producer and consumer clients in the kafka-connect.log .. am I just misconfiguring this then?

On Wednesday, August 2, 2017 at 12:44:47 PM UTC-7, Ewen Cheslack-Postava wrote:

That class is only for the broker. org.apache.kafka.common.metrics.MetricsReporter is the one you'd want to use. It will work with everything that ships with AK (i.e. Kafka, Java clients, Connect, Streams). Schema Registry, Rest Proxy and other Confluent products use a different, but essentially the same interface: https://github.com/confluentinc/common/blob/master/metrics/src/main/java/io/confluent/common/metrics/MetricsReporter.java

On Tue, Aug 1, 2017 at 1:53 PM, Eric Pheatt <eric....@gmail.com> wrote:

Should the kafka.metrics.reporters for the satellite confluent services such as connect, rest-proxy and schema-registry work out of the box or is the KafkaMetricsReporter interface just intended for broker metrics?

I'm evaluating the graphite reporter provided by https://github.com/apakulov/kafka-graphite and it is working well enough for monitoring the broker but it doesn't look like the other confluent services participate directly with that Reporter framework. 

Thanks,

Eric

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

[Eric Pheatt]

I'll dig deeper into the connect init process but it doesn't look like the metrics.reporters is being set in the consumer/producer clients in the logs for connect. Is there something else I should be looking for in the logs to indicate that it has been correctly picked up? In the short term I've adapted a simple poll based approach based on a a process I used to use for Solr and Redis monitoring under a personal repo https://github.com/epheatt/mvkg but I'd much rather have the push based MetricsReporter since we are planning to run the connect instances as part of a containerized spot fleet in AWS and cron will be a pain in a container.

[Eric Pheatt]

Ahh, I see the issue with my config, in the worker.properties it should have been specified as metric.reporters instead of kafka.metrics.reporters for the client. That said, when I do that the process fails to start with the following error .. so it looks like this reporter implimentation isn't ready for prime time yet for client metrics.

Exception in thread "main" org.apache.kafka.common.KafkaException: Failed to construct kafka consumer

        at org.apache.kafka.connect.runtime.distributed.WorkerGroupMember.<init>(WorkerGroupMember.java:128)

        at org.apache.kafka.connect.runtime.distributed.DistributedHerder.<init>(DistributedHerder.java:165)

        at org.apache.kafka.connect.runtime.distributed.DistributedHerder.<init>(DistributedHerder.java:145)

        at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:80)

Caused by: org.apache.kafka.common.KafkaException: Could not instantiate class org.apache.kafka.common.metrics.MetricsReporter Does it have a public no-argument constructor?

        at org.apache.kafka.common.utils.Utils.newInstance(Utils.java:342)

        at org.apache.kafka.common.utils.Utils.newInstance(Utils.java:356)

        at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstances(AbstractConfig.java:243)

        at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstances(AbstractConfig.java:220)

        at org.apache.kafka.connect.runtime.distributed.WorkerGroupMember.<init>(WorkerGroupMember.java:86)

        ... 3 more

Caused by: java.lang.InstantiationException: org.apache.kafka.common.metrics.MetricsReporter

        at java.lang.Class.newInstance(Class.java:427)

        at org.apache.kafka.common.utils.Utils.newInstance(Utils.java:338)

        ... 7 more

Caused by: java.lang.NoSuchMethodException: org.apache.kafka.common.metrics.MetricsReporter.<init>()

        at java.lang.Class.getConstructor0(Class.java:3082)

        at java.lang.Class.newInstance(Class.java:412)

        ... 8 more

[Eric Pheatt]

Posted too quickly .. nothing to see here .. the following works as expected..

metric.reporters=org.apache.kafka.common.metrics.GraphiteReporter

[Eric Pheatt]

For confluent schema-registry support I hacked a quick implementation together here https://github.com/epheatt/kafka-graphite/commit/6bbdaa to use the io.confluent.common.metrics.GraphiteReporter and just switched around the org.apache.kafka.common.metrics.* class over to io.confluent.common.metrics.*. While I see the class registered on start up of the schema registry in the log in the main server

[2017-08-04 16:54:31,380] INFO SchemaRegistryConfig values: 

        metric.reporters = [io.confluent.common.metrics.GraphiteReporter]

...

[2017-08-04 16:54:31,455] INFO GraphiteConfig values: 

        kafka.graphite.metrics.exclude = 

        kafka.graphite.metrics.host = graphite.host

        kafka.graphite.metrics.include = 

        kafka.graphite.metrics.port = 2003

        kafka.graphite.metrics.prefix = collectd.shared.KAFKAS.shared-kfks-1

        kafka.graphite.metrics.reporter.enabled = true

        kafka.metrics.polling.interval.secs = 60

 (io.confluent.common.metrics.GraphiteReporter$GraphiteConfig:180)

[2017-08-04 16:54:31,465] INFO Configuring Kafka Graphite Reporter with host=graphite.host, port=2003, prefix=collectd.shared.KAFKAS.shared-kfks-1 and include=, exclude= (io.confluent.common.metrics.GraphiteReporter:86)

I don't see the metric.reporters set for the producers or consumer clients. If I set the metric.reporters=org.apache.kafka.common.metrics.GraphiteReporter in the schema-registry.properties it throws an error expecting io.confluent.common.metrics.MetricsReporter vs org.apache.kafka.common.metrics.MetricsReporter.

How can I get both metrics reporters configured for the main process and the client processes at the same time with different classes?

Thanks,

Eric