MongoDB-Kafka Implementation

[Dileep D]

Hi, 

I am trying to leverage mongo DB connector for Kafka ,https://www.mongodb.com/blog/post/getting-started-with-the-mongodb-connector-for-apache-kafka-and-mongodb-atlas

For sink part , I am getting error as 

Invalid value io.confluent.connect.avro.AvroConverter for configuration key.converter: Class io.confluent.connect.avro.AvroConverter could not be found.

Invalid value io.confluent.connect.avro.AvroConverter for configuration value.converter: Class io.confluent.connect.avro.AvroConverter could not be found.

I am not using Confluent package but using freely available kafka.

Is this connector will work only with confluent Package. How to avoid this error . How to use avro in normal apache kafka.

Please do let me know

[Wan Bachtiar]

Is this connector will work only with confluent Package. How to avoid this error . How to use avro in normal apache kafka.

Hi Dileep,

The MongoDB Connector for Kafka can be used with either Confluent’s Kafka or Apache’s Kafka deployments. The Confluent’s Kafka has some additional tooling that is beneficial to enterprise and production deployments of Kafka.

The error message that you’re getting indicates that it could not find the Avro converter specified for key.converter and value.converter configuration. Also worth mentioning that the Avro converter makes use of Confluent’s schema registry, so if you don’t have the registry it would be difficult to setup.

Depending on the use case, you can also use formats other than Avro. The sink connector is configurable to support other data formats, such as JSON with schema, JSON plain and RAW JSON. i.e. org.apache.kafka.connect.json.JsonConverter. See also MongoDB Kafka sink connector guide.

If you have further questions about the Avro converter without Confluent Schema Registry I would suggest to post a question on Kafka forums to reach wider audience with the expertise.

Regards,
Wan.

​