Debizium Mysql Connector error

2,386 views
Skip to first unread message

avi Singh

unread,
Nov 20, 2017, 4:00:11 PM11/20/17
to debezium
Guys can anyone please help me with this error below

Mysql 5.7
Kafka connect: 0.11.0.2
Debezium: 0.6.2

I have updated the connect-standalone.properties file with mysql connector plugin 
plugin.path=/usr/local/share/kafka/plugins/debezium-connector-mysql


Now when i try to start it throws the error below

[root@ip-172-31-47-198 kafka_2.11-0.11.0.2]# bin/connect-standalone.sh config/connect-standalone.properties config/connect-file-sink.properties
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
[2017-11-20 20:12:22,494] INFO Loading plugin from: /usr/local/share/kafka/plugins/debezium-connector-mysql/debezium-connector-mysql-0.6.2.jar (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:176)
Exception in thread "main" java.lang.NoClassDefFoundError: io/debezium/util/IoUtil
at io.debezium.connector.mysql.Module.<clinit>(Module.java:19)
at io.debezium.connector.mysql.MySqlConnector.version(MySqlConnector.java:46)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.getPluginDesc(DelegatingClassLoader.java:242)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:223)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:198)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:190)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:150)
at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:47)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:68)
Caused by: java.lang.ClassNotFoundException: io.debezium.util.IoUtil
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at org.apache.kafka.connect.runtime.isolation.PluginClassLoader.loadClass(PluginClassLoader.java:62)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 9 more


Thanks
 avi

Randall Hauch

unread,
Nov 20, 2017, 6:24:47 PM11/20/17
to debezium
On Mon, Nov 20, 2017 at 3:00 PM, avi Singh <avisingh...@gmail.com> wrote:
Guys can anyone please help me with this error below

Mysql 5.7
Kafka connect: 0.11.0.2
Debezium: 0.6.2

I have updated the connect-standalone.properties file with mysql connector plugin 
plugin.path=/usr/local/share/kafka/plugins/debezium-connector-mysql


Try

plugin.path=/usr/local/share/kafka/plugins/

The plugin path must contain the directories that contain the plugin directories (e.g., "debezium-connector-mysql"). 


Now when i try to start it throws the error below

[root@ip-172-31-47-198 kafka_2.11-0.11.0.2]# bin/connect-standalone.sh config/connect-standalone.properties config/connect-file-sink.properties
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
[2017-11-20 20:12:22,494] INFO Loading plugin from: /usr/local/share/kafka/plugins/debezium-connector-mysql/debezium-connector-mysql-0.6.2.jar (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:176)
Exception in thread "main" java.lang.NoClassDefFoundError: io/debezium/util/IoUtil
at io.debezium.connector.mysql.Module.<clinit>(Module.java:19)
at io.debezium.connector.mysql.MySqlConnector.version(MySqlConnector.java:46)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.getPluginDesc(DelegatingClassLoader.java:242)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:223)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:198)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:190)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:150)
at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:47)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:68)
Caused by: java.lang.ClassNotFoundException: io.debezium.util.IoUtil
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at org.apache.kafka.connect.runtime.isolation.PluginClassLoader.loadClass(PluginClassLoader.java:62)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 9 more


Thanks
 avi

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+unsubscribe@googlegroups.com.
To post to this group, send email to debe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/d2034b04-3f19-427d-8e89-7debf0f893df%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

avi Singh

unread,
Nov 21, 2017, 1:55:52 PM11/21/17
to debezium
Thank you, that worked but now it is throwing the below listed errors when i try to register configuration file using this command. 

bin/connect-standalone.sh config/connect-standalone.properties config/mysql.properties config/connect-file-sink.properties


[root@ip-172-31-47-198 config]# more mysql.properties
{
 "name": "connect-mysql",
 "config":
 {
   "connector.class": "io.debezium.connector.mysql.MySqlConnector",
   "database.hostname": "172.31.47.198",
   "database.port": "3306",
   "database.user": "debezium",
   "database.password": "MyNewPass4!",
   "database.server.name": "ip-172-31-47-198.us-east-2.compute.internal",
   "database.whitelist": "debezium_test",
   "include.schema.changes"  : "false"
 }


[2017-11-21 18:53:03,853] INFO Added alias 'ValueToKey' to plugin 'org.apache.kafka.connect.transforms.ValueToKey' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:290)
Exception in thread "main" org.apache.kafka.common.config.ConfigException: Missing required configuration "key.converter" which has no default value.
at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:463)
at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:453)
at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:62)
at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:75)
at org.apache.kafka.connect.runtime.WorkerConfig.<init>(WorkerConfig.java:197)
at org.apache.kafka.connect.runtime.standalone.StandaloneConfig.<init>(StandaloneConfig.java:42)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:70)

Randall Hauch

unread,
Nov 21, 2017, 5:25:23 PM11/21/17
to debezium
You're not setting all of the properties required by the standalone worker. Please see https://docs.confluent.io/current/connect/userguide.html#configuring-workers

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+unsubscribe@googlegroups.com.
To post to this group, send email to debe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/a36cd5d9-9daf-4922-94f8-b428951744fc%40googlegroups.com.

avi Singh

unread,
Nov 21, 2017, 7:29:43 PM11/21/17
to debezium
Thanks a lot Randall. i followed the link and made the necessary changes to the file but i am still getting some errors. This is how my files look like now. Can you please look at the errors.

[root@ip-172-31-47-198 config]# more connect-standalone.properties

bootstrap.servers=localhost:9092
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=true
value.converter.schemas.enable=true
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
offset.storage.file.filename=/tmp/connect.offsets
plugin.path=/usr/local/share/kafka/plugins


[root@ip-172-31-47-198 config]# more connect-file-source.properties

name=debezium_test-connector
connector.class=io.debezium.connector.mysql.MySqlConnector
tasks.max= 1
database.hostname=172.31.47.198
database.port=3306
database.user=debezium
database.password=password
database.server.name=ip-172-31-47-198.us-east-2.compute.internal
database.whitelist=debezium_test
database.history.kafka.bootstrap.servers=kafka:9092
database.history.kafka.topic=internal


Now when i run this below command i get these following errors. 

[root@ip-172-31-47-198 kafka_2.11-0.11.0.2]# bin/connect-standalone.sh config/connect-standalone.properties config/connect-file-source.properties

[2017-11-22 00:24:56,191] WARN Removing server kafka:9092 from bootstrap.servers as DNS resolution failed for kafka (org.apache.kafka.clients.ClientUtils:54)
[2017-11-22 00:24:56,191] INFO Closing the Kafka producer with timeoutMillis = 0 ms. (org.apache.kafka.clients.producer.KafkaProducer:1017)
[2017-11-22 00:24:56,191] INFO Stopping MySQL connector task (io.debezium.connector.mysql.MySqlConnectorTask:220)
[2017-11-22 00:24:56,192] ERROR Task debezium_test-connector-0 threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:148)
org.apache.kafka.connect.errors.ConnectException: org.apache.kafka.common.KafkaException: Failed to construct kafka producer
at io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:194)
at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:142)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:146)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:190)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka producer
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:416)
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:288)
at io.debezium.relational.history.KafkaDatabaseHistory.start(KafkaDatabaseHistory.java:150)
at io.debezium.connector.mysql.MySqlSchema.start(MySqlSchema.java:150)
at io.debezium.connector.mysql.MySqlTaskContext.start(MySqlTaskContext.java:202)
at io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:67)
... 8 more
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:64)
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:373)
... 13 more
[2017-11-22 00:24:56,194] ERROR Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:149)










To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.

To post to this group, send email to debe...@googlegroups.com.

Randall Hauch

unread,
Nov 22, 2017, 10:28:55 AM11/22/17
to debezium
I think this is fairly obvious ... the bootstrap URLs for the Kafka producer are not resolvable. There are two places in your configs where you specify Kafka bootstrap URLs:

    bootstrap.servers=localhost:9092

and

   database.history.kafka.bootstrap.servers=kafka:9092

Note how those two URLs don't match? I suspect they should. So which one is wrong? Looking at the stack trace, this error is happening within the MySQL connector's `KafkaDatabaseHistory`, which suggests that it's the latter property that's the problem. Try changing it to

   database.history.kafka.bootstrap.servers= localhost:9092

 
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+unsubscribe@googlegroups.com.

To post to this group, send email to debe...@googlegroups.com.

avi Singh

unread,
Nov 22, 2017, 1:36:59 PM11/22/17
to debezium
Thanks a lot Randall. It's working now. 

avi Singh

unread,
Nov 22, 2017, 6:01:04 PM11/22/17
to debezium
Randall can u please help me again with avro issue. i started zookeeper/kafka/schema registry. i made changes to connect-standalone.properties file by adding avo parameters

more connect-standalone.properties

bootstrap.servers=localhost:9092
key.converter=io.confluent.connect.avro.AvroConverter
key.converter.schema.registry.url=http://localhost:8081
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=http://localhost:8081
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
offset.storage.file.filename=/tmp/connect.offsets
plugin.path=/usr/local/share/kafka/plugins


When i run below listed command i get these errors

[root@ip-172-31-47-198 kafka_2.11-0.11.0.2]# bin/connect-standalone.sh config/connect-standalone.properties config/connect-file-source.properties

[2017-11-22 22:54:03,535] INFO Added alias 'ValueToKey' to plugin 'org.apache.kafka.connect.transforms.ValueToKey' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:290)
Exception in thread "main" org.apache.kafka.common.config.ConfigException: Invalid value io.confluent.connect.avro.AvroConverter for configuration key.converter: Class io.confluent.connect.avro.AvroConverter could not be found.
at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:715)
at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:460)
at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:453)
at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:62)
at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:75)
at org.apache.kafka.connect.runtime.WorkerConfig.<init>(WorkerConfig.java:197)
at org.apache.kafka.connect.runtime.standalone.StandaloneConfig.<init>(StandaloneConfig.java:42)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:70)
On Wednesday, November 22, 2017 at 7:28:55 AM UTC-8, Randall Hauch wrote:

Gunnar Morling

unread,
Nov 23, 2017, 5:18:02 AM11/23/17
to debezium
kafka-connect-avro-converter-x.y.z.jar is missing in your Connect instance. See the Docker file of our Connect Base image for the full list of dependencies you should have:



--Gunnar
Reply all
Reply to author
Forward
0 new messages