running Debezium 1.2 Cassandra connector

[srisumi]

I am trying to run debezium cassandra connector on my laptop for a POC. I am following https://debezium.io/documentation/reference/1.2/connectors/cassandra.html#cassandra-deploying-a-connector but don't find a way to start the connector as a Java process. I am running cassandra, Kafka and mysql locally. How do I run the connector (v1.2) as a standalone java process? I am not using docker.

Thanks,

Sri

[Ahmed Eljami]

Hi,

first, you should download the jar from the maven repo: https://repo1.maven.org/maven2/io/debezium/debezium-connector-cassandra/

Then, run the following command to start the java process:

java -jar debezium-connector-cassandra-X.X.X.Final-jar-with-dependencies.jar cdc.properties > debezium.stdout.log 2> debezium.stderr.log

cdc.properties example:

connector.name=test_connector

commit.log.relocation.dir=/tmp/debezium/relocation/

http.port=8000

cassandra.config=/etc/cassandra/cassandra.yaml

cassandra.hosts=cassandra-seed

cassandra.port=9042

kafka.producer.bootstrap.servers=kafka1:9092

kafka.producer.retries=3

kafka.producer.retry.backoff.ms=1000

key.converter=io.confluent.connect.avro.AvroConverter

key.converter.schema.registry.url: http://schemaregistry:8081

value.converter=io.confluent.connect.avro.AvroConverter

value.converter.schema.registry.url: http://schemaregistry:8081

offset.backing.store.dir=/tmp/debezium/offsets/

snapshot.consistency=ONE

Hope this help you.

Cheers

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/79a8646f-dc4f-4f8f-981a-ee3807bbb0f7n%40googlegroups.com.

--

Cordialement;

Ahmed ELJAMI

[srisumi]

Thanks for your response. I am running DSE (datastax) Cassandra and the libraries that come with that distro. 

And obviously, the debezium-connector-cassandra-1.2.0.Final-jar-with-dependencies.jar is not able to find the DSE jars in the classpath. 

For example, the debezium connector exits with: java.lang.ClassNotFoundException: com.datastax.bdp.snitch.DseSimpleSnitch

How can I run debezium connector with DSE provided jars as dependencies?

[Ahmed Eljami]

Hey,

Not sure if the connector supports DSE, I have encountered an issue with the snitch property that I have "resolved" (a workaround) by setting the snitch to SimpleSnitch, described here: https://issues.redhat.com/browse/DBZ-1987

In your case, it seems like DSE does not support this snitch. Anyway, it's not mentioned that Debezium support DSE.... :)

Cheers

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/db60cd24-5ebc-43a0-a45c-ef39b8e3383en%40googlegroups.com.

--

Cordialement;

Ahmed ELJAMI

[srisumi]

Thanks for this info. I switched to using the open source Cassandra and this time around the debezium connector came up but stopped with the following error. 

I can't seem to find the cause. I have attached the full log file.

17:33:37.013 [pool-4-thread-5] ERROR i.d.c.c.CassandraConnectorTask - Encountered exception while running Commit Log Processor; stopping all processors in Cassandra Connector Task
java.lang.InterruptedException: null
at io.debezium.connector.cassandra.CommitLogProcessor.process(CommitLogProcessor.java:83) ~[debezium-connector-cassandra-1.2.2.Final-jar-with-dependencies.jar:1.2.2.Final]
at io.debezium.connector.cassandra.AbstractProcessor.start(AbstractProcessor.java:61) ~[debezium-connector-cassandra-1.2.2.Final-jar-with-dependencies.jar:1.2.2.Final]
at io.debezium.connector.cassandra.CassandraConnectorTask$ProcessorGroup.lambda$start$0(CassandraConnectorTask.java:200) ~[debezium-connector-cassandra-1.2.2.Final-jar-with-dependencies.jar:1.2.2.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_201]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_201]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_201]
17:33:37.013 [pool-4-thread-5] INFO i.d.c.cassandra.AbstractProcessor - Stopping Schema Processor
17:33:37.013 [pool-4-thread-5] INFO i.d.c.cassandra.AbstractProcessor - Stopping Change Event Queue Processor
17:33:37.013 [pool-4-thread-5] INFO o.a.k.clients.producer.KafkaProducer - [Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.

[Gunnar Morling]

Hi,

Hum, that's interesting. Looks like the connector actually was stopped. Did you press Ctrl+C by any chance? Re DSE, it's not tested and I'm not sure what it'd entail to support it. Any insights on that will be much appreciated.

--Gunnar

[Ahmed Eljami]

Hi,

I advise you to use the latest version of dbz-connector, some issuers have been resolved since.

If you encountered the same issue, it would be better if you give us more details : cassandra version, steps to reproduce the issue

Cheers

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/0020d3e6-ad36-4567-a139-07a7b527b36an%40googlegroups.com.

--

Cordialement;

Ahmed ELJAMI

[srisumi]

1. I am running:

debezium-connector-cassandra-1.2.2.Final-plugin.tar.gz
apache-cassandra-3.11.8
kafka_2.13-2.6.0

2. I have attached the debezium connect.properties, zookeeper.properties, kafka server.properties & cassandra.yaml files.

3. Here are the start commands:

  bin/zookeeper-server-start.sh config/zookeeper.properties  

  bin/kafka-server-start.sh config/server.properties

  java -jar debezium-connector-cassandra-1.2.2.Final-jar-with-dependencies.jar /Users/srchakra/Downloads/debezium/conf/connect.properties

  

Please let me know if you need any other info to reproduce this issue.

thanks

Sri

[Bingqin Zhou]

Looks like you set latest.commit.log.only=true in your connector.properties file. This config should be set to false in your case, otherwise, the connector is expected to stop after having processed the latest commitlog file in cassandra.

--

You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/cfd929e5-31f5-4706-9445-9b93bd41f8e2n%40googlegroups.com.

[Cyril Scetbon]

I’m surprised that in the logs we see 

ERROR i.d.c.c.CassandraConnectorTask - Encountered exception while running Commit Log Processor; stopping all processors in Cassandra Connector Task 
java.lang.InterruptedException: null 

It should be clear that it stops because of that parameter.Seeing an exception is misleading, what do you think ?

—

Cyril Scetbon

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/CABYXsiGFZb%2BWFEP1_XBMYj%3D3%3DgC3o5DJN7_jDFku%3D%2BU8N-TQnA%40mail.gmail.com.

[Qianji Zheng]

I am doing a POC using DSE and Cassandra connector. Any insights on that will be much appreciated.