Kafka Connect - Custom MetricReporter ClassNotFoundException

[Matt Veitas]

Good morning,

I am trying to add a custom MetricReporter to a running Kafka Connect app (running in distributed mode) which is running in Docker. I have created a custom class: "CustomMetricsReporter" and have set the environment variable in Docker to the following: CONNECT_METRIC_REPORTERS: "com.everbridge.kafka.connect.metrics.CustomMetricsReporter". When starting the connector, we get a ClassNotFoundException being thrown as it can't seem to find the class file (stack trace below).

Where should the jar file with the MetricReporter implementation be placed? I have tried a few places such as "/usr/share/java/kafka", "/usr/share/java/confluent-common", and within the plugin path (didn't expect this to work).

Associated stack trace

------------------------------------------------------------------------------------------------------

[main] ERROR io.confluent.admin.utils.cli.KafkaReadyCommand - Error while running kafka-ready.

org.apache.kafka.common.KafkaException: Failed create new KafkaAdminClient

        at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:338)

        at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:62)

        at io.confluent.admin.utils.ClusterStatus.isKafkaReady(ClusterStatus.java:138)

        at io.confluent.admin.utils.cli.KafkaReadyCommand.main(KafkaReadyCommand.java:150)

Caused by: org.apache.kafka.common.KafkaException: com.everbridge.kafka.connect.metrics.LoggingMetricsReporter ClassNotFoundException exception occurred

        at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstances(AbstractConfig.java:357)

        at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstances(AbstractConfig.java:332)

        at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstances(AbstractConfig.java:319)

        at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:304)

        ... 3 more

Caused by: java.lang.ClassNotFoundException: com.everbridge.kafka.connect.metrics. CustomMetricsReporter

        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

        at java.lang.Class.forName0(Native Method)

        at java.lang.Class.forName(Class.java:348)

        at org.apache.kafka.common.utils.Utils.loadClass(Utils.java:322)

        at org.apache.kafka.common.utils.Utils.newInstance(Utils.java:311)

        at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstances(AbstractConfig.java:355)

-Matt

[Matt Veitas]

Bumping this...

[Konstantine Karantasis]

Can you please mention which version of the Confluent Platform and Kafka Connect are you using?

Also this is the complete stacktrace? 

Did you try to add the classes of the custom metrics reporter in the CLASSPATH as well?

Konstantine

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/6fa916eb-f4e8-4c88-9ae2-84cd087bdcb0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matt Veitas]

Hi Konstantine,

We are currently using Confluent 4.1.0. I'll double check on the exception, but I think that was the entire stack trace. I'll reproduce this tonight and confirm.

What do you mean by adding the classes of the custom metrics reporter in the "CLASSPATH" as well? Are you referring to the environment variable that is used in the kafka_run.sh? I tried to set that in my docker configuration as an environment variable, but it was never picked up on.

Am I right that the jar that contains the class needs to be placed somewhere the connector can get access to, and not the classpath for the connector plugins, correct?

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

[mag...@confluent.io]

Hi Matt,

It's the cub utility that's failing. Cub utility checks if the target Kafka cluster is ready. In the process, it uses the same configuration file as connect. But the classpath used by Bud utility is different compared to the connect classpath. So, to get this metrics reporter to work,

you need to also add the jar to /usr/share/java/cp-base/* as well. Another option you try out is to add a CUB_CLASSPATH environment variable that includes /usr/share/java/cp-base/* and also the path to the metrics jar. Let me know if it helps.

Thanks

Magesh

[Matt Veitas]

Magesh,

Awesome. Now that I look at the stack trace that does make sense and didn't jump out at me right away. I'll try this out and reply back. Assuming this works it might be worthwhile to add some notes into the documentation

Thanks again in advance to looking at this!

-Matt

[Matt Veitas]

No luck with the suggestions as /usr/share/java/cp-base does not exist in the image.  

When I start up the app, I do see that CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar is already referenced which seems to contain the files used to help run the utilities.

-Matt

[Magesh Nandakumar]

Yes, that's right. CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar is the right one. It's already defined in the base docker file. As it is, I don't see an option to pass additional paths to CUB_CLASSPATH. Could you maybe try to build an image with the existing image as the base and set the CUB_CLASSPATH in there? I haven't tried this out but an option to explore.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/6fa916eb-f4e8-4c88-9ae2-84cd087bdcb0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/ygFwDxQTrfc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/827b96da-4a0b-4e7e-9da5-52ebd7eb6e64%40googlegroups.com.

[Matt Veitas]

I ended up getting it to by just setting the CUB_CLASSPATH to the right value in the docker environment variable (it was a fat finger typo on my part that was preventing it from working). This is a bit fragile as we now have this hardcoded path of /etc/confluent/docker/docker-utils.jar.

Thank you very much for the assistance!

Matt

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/6fa916eb-f4e8-4c88-9ae2-84cd087bdcb0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/ygFwDxQTrfc/unsubscribe.

To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.