AWS MSK connector is not getting started with Cassandra connector

[Yogesh Patil]

Hello there,

I've created AWS MSK connector for Cassandra but somehow it is not getting started due to below error-

Caused by: org.apache.cassandra.exceptions.ConfigurationException: Cannot locate file:///usr/local/cassandra/conf/cassandra.yaml.  If this is a local file, please confirm you've provided file:/// as a URI prefix.
[Worker-03d4c8b8e21cae672] at org.apache.cassandra.config.YamlConfigurationLoader.getStorageConfigURL(YamlConfigurationLoader.java:89)

Is this because of MSK is assuming it would be at local path?

Things to note-

1. My Cassandra 3.10 is up & running with cdc_enabled at nodes as well table level

2. cdc_raw_directory is configured in cassandra.yaml

3. AWS MSK is configured

4. Both Cassandra and MSK are running on different instances.

5. Have modified inbound/outboud rules to allow communication between cassandra & msk cluster

Connector config-

{
  "connector.class": "io.debezium.connector.cassandra.Cassandra3Connector",
  "keyspace.include.list": "ncl",
  "tasks.max": "1",
  "cassandra.ssl.enabled": "false",
  "cassandra.username": "cassandra",
  "cassandra.hosts": "hostname",
  "cassandra.port": "9042",
  "schema.history.internal.kafka.bootstrap.servers": "brokers",
  "offset.flush.interval.ms": "60000",
  "key.converter.schemas.enable": "true",
  "topic.prefix": "cassandra.cdc",
  "kafka.bootstrap.servers": "brokers",
  "schema.history.internal.kafka.topic": "schema-history.cassandra",
  "commit.log.real.time.processing.enabled": "true",
  "value.converter.schemas.enable": "true",
  "cassandra.password": "cassandra",
  "table.include.list": "ncl.cdc_demo_events",
  "value.converter": "org.apache.kafka.connect.json.JsonConverter",
  "commit.log.relocation.dir": "/usr/local/cassandra/data/cdc_raw/",
  "key.converter": "org.apache.kafka.connect.json.JsonConverter",
  "snapshot.mode": "initial",
  "cassandra.config": "/usr/local/cassandra/conf/cassandra.yaml"
}

[jiri.p...@gmail.com]

Hi,

Cassandra connector is different form other connctors as it does not run in Kafka Connect but side-by-side with Cassandra nodes - see https://debezium.io/documentation/reference/3.1/connectors/cassandra.html#cassandra-deploying-a-connector

Jiri

[Yogesh Patil]

Thanks for the suggestion Jiri. Will give it a try to run debezium across all the cassandra nodes and see if it is able to push db mutations (of cdc enabled tables) to the configured kafka topic.

However, I want to validate my understanding on this. Once debezium jar is up & running across all the Cassandra nodes with required config (brokers, etc), on any db mutations it reads commitLogs from cdc directory and pushes it to Kafka topic?

Also, is AWS MSK connector is not at all valid CDC design approach (for cassandra) as it is altogether running in different space (as one of the AWS services) and can not connect and read distributed cassandra nodes commit logs and push it to its own Kafka brokers (in msk cluster)?

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/debezium/c6978fff-d26f-4867-8a5b-97fe93d3e6b6n%40googlegroups.com.

--

Thanks,

Yogesh Patil

[jiri.p...@gmail.com]

Hi,

AWS MSK Connect is valid approach for all Debezium connectors but Cassandra. ASW MSK as such is valid solution for an connector as it is effectivewlly just managed Kafka broker cluster.

Jiri