Kafka and Debezium help needed (ASAP)

[Partha Sarathi Chakravarty]

Hello, I am intern and have been tasked to configure Debezium MySql connector with Kafka for capturing CDC without using docker from the local MySql server on ubuntu(virtualised). I have already downloaded and set the 'plugin.path' for the debezium .jar files and my Kafka and zookeeper is up and running. But whenever I try starting kafka in the standalone-mode it initialises the debezium connector and comes to a halt and the process gets terminated. I have tried numerous attempts but all were unsuccessful. I have also tried following articles on the internet but all are using dockereized container and mostly postgres.

It would be great if I could get detailed help from anyone to get the setup up and running as kafka is a bit new environment for me.

[jiri.p...@gmail.com]

Hi,

please take a look at plugin.path config option. Find the directories and make sure there is a subdirectory debezium-connector-mysql that was installed via https://debezium.io/documentation/reference/2.0/install.html

J.

[Nathan Smit]

Have you considered using Debezium Server to eliminate the need for Kafka if you're unfamiliar with Kafka and Zookeeper?  You can just download the bundled debezium server and you just run it from a shell script.  In any event, I previously documented the following steps for running Debezium in our environment.  We're running Oracle so just ignore oracle related stuff.  It's also from a while back, but maybe the more detailed steps here will help.

Current setup steps:

1. Untar the downloaded kafka installation

2. Under /config/ create a “connectors” folder.  This is where connector jar is currently placed

3. Download the Debezium connector for Oracle from here:  https://debezium.io/releases/1.6/

4. Config changes to Kafka:

   1. Under config/connect-standalone.properties, I’ve changed plugin path to:  plugin.path=/home/dba/kafka_2.12-2.8.1/config/connectors (connectors directory added in previous step - todo - make this more generic)

   2. Under config/zookeeper.properties I’ve changed the dataDir to dataDir=/home/dba/kafka_2.12-2.8.1/zookeeper-logs - todo - change config to get other logging/files out of the tmp directory

   3. Under /config/  created a new properties file called debezium-oracle-connector.  Content of this file is contained in the addendum.

5. Run following commands from the kafka base directory to get everything up and running:

   1. Start Zookeeper:  bin/zookeeper-server-start.sh config/zookeeper.properties

   2. Start Kafka:  bin/kafka-server-start.sh config/server.properties

   3. Start Connector:  bin/connect-standalone.sh config/connect-standalone.properties config/debezium-oracle-connector.properties

6. To view all of the kafka topics, run the following from kafka directory:  bin/kafka-topics.sh --list --bootstrap-server localhost:9092 

7. To consume all message from the topic, run the following from kafka directory:  bin/kafka-console-consumer.sh     --bootstrap-server localhost:9092     --from-beginning     --property print.key=true     --topic debezium.LOOKUP.BRANCH_TEST

[Partha Sarathi Chakravarty]

Thank you very much for the suggestions. I will try all of it and update you for the same.

--
You received this message because you are subscribed to a topic in the Google Groups "debezium" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/debezium/vjCYhCUMMK0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/bbeaacc7-dd3f-4614-a8a4-2d96aa5d2cb5n%40googlegroups.com.

[Partha Sarathi Chakravarty]

Well I had worked on the solution but unfortunately nothing happened. Still unable to start it in a standalone mode.

Is there any other solution you can suggest? I just want to get the setup running.

[jiri.p...@gmail.com]

Are you sure that that there is a file name /config/connect-standalone.properties ? Note the starting slash char

J.

[Partha Sarathi Chakravarty]

Yes Sir I am pretty sure. I am sending you a screenshot for the same where I have a file in the location 'home/plinux/kafka_2.13-3.3.1/config/connect-standalone.properties'. It is the same place where I have configured my 'plugin.path' with the 'debezium-connector-mysql' .jar location.

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/2738c79e-1907-4a8a-832e-9c2e31609accn%40googlegroups.com.

[jiri.p...@gmail.com]

COuld you please share the command you use to start Kafka Connect?

J.

[Partha Sarathi Chakravarty]

Yes, of course.

The command is :
kafka 2.13-3.3.1/bin/connect-standalone.sh kafka 2.13-3.3.1/config/connect-standalone.properties kafka 2.13-3.3.1/config/connect-debezium-mysql.properties

I intended to start Kafka in the standalone mode, I have also specified the folder for the debezium .jar files at the end of the command. I am attaching a screenshot for your reference.

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/5438e059-1e13-4dc6-94db-b5b3b7b366can%40googlegroups.com.