While testing the recently installed Confluent-Kafka platform, we are able to produce and consume Avro messages using kafka-avro-console-producer and kafka-avro-console-consumer without any issue. But while using connect-standalone command in Kafka-connect-hdfs, we are facing below error:
ERROR Thread WorkerSinkTask-hdfs-sink-valent-0 exiting with uncaught exception: (org.apache.kafka.connect.util.ShutdownableThread:84)
java.lang.NoSuchMethodError: org.apache.kafka.clients.consumer.KafkaConsumer.subscribe(Ljava/util/List;Lorg/apache/kafka/clients/consumer/ConsumerRebalanceListener;)V
at org.apache.kafka.connect.runtime.WorkerSinkTask.joinConsumerGroupAndStart(WorkerSinkTask.java:143)
at org.apache.kafka.connect.runtime.WorkerSinkTaskThread.execute(WorkerSinkTaskThread.java:54)
at org.apache.kafka.connect.util.ShutdownableThread.run(ShutdownableThread.java:82)
Exception in thread "WorkerSinkTask-hdfs-sink-valent-0" java.lang.NoSuchMethodError: org.apache.kafka.clients.consumer.KafkaConsumer.subscribe(Ljava/util/List;Lorg/apache/kafka/clients/consumer/ConsumerRebalanceListener;)V
at org.apache.kafka.connect.runtime.WorkerSinkTask.joinConsumerGroupAndStart(WorkerSinkTask.java:143)
at org.apache.kafka.connect.runtime.WorkerSinkTaskThread.execute(WorkerSinkTaskThread.java:54)
at org.apache.kafka.connect.util.ShutdownableThread.run(ShutdownableThread.java:82)
The details are:
OS: centos 6
Java version: 1.7.x
Confluent-kafka version: confluent-platform-2.11
Versions of components:
========================================================================================================================================================================
Package Arch Version Repository Size
========================================================================================================================================================================
Installing:
confluent-platform-2.11 noarch 3.0.0-1 Confluent 6.6 k
Installing for dependencies:
confluent-camus noarch 3.0.0-1 Confluent 19 M
confluent-common noarch 3.0.0-1 Confluent 2.0 M
confluent-control-center noarch 3.0.0-1 Confluent 34 M
confluent-kafka-2.11 noarch 0.10.0.0-1 Confluent 33 M
confluent-kafka-connect-hdfs noarch 3.0.0-1 Confluent 86 M
confluent-kafka-connect-jdbc noarch 3.0.0-1 Confluent 5.9 M
confluent-kafka-rest noarch 3.0.0-1 Confluent 16 M
confluent-rest-utils noarch 3.0.0-1 Confluent 7.6 M
confluent-schema-registry noarch 3.0.0-1 Confluent 26 M
confluent-support-metrics noarch 3.0.0-1 Confluent 43 k
The /etc/kafka/connect-file-source.properties:
name=hdfs-sink-valent
connector.class=io.confluent.connect.hdfs.HdfsSinkConnector
tasks.max=1
topics=valent
hdfs.url=hdfs://xxx.yyy.zzz.ppp:8020
flush.size=1
hive.integration=true
schema.compatibility=BACKWARD
format.class=io.confluent.connect.hdfs.parquet.ParquetFormat
Any Idea?