Kafka Connect - between two Kafka topics..

3,624 views
Skip to first unread message

Saravanan Tirugnanum

unread,
May 27, 2016, 10:38:39 AM5/27/16
to Confluent Platform
I have to transport messages between two Kafka Topics and do some transformation before publishing. Understand Kafka Streams will fit this requirement but we cannot migrate to 0.10 Kafka Cluster in near future.. So , my thoughts are to use Kafka Connect Sink Job to move messages between Kafka topics.

We have to build multiple connectors with different sources with Kafka also being one of the source before publishing to final Kafka topic for further processing . 
So , Can Connect  be an option ..

Is this a good approach . Any thoughts..

Regards
Saravanan

Ewen Cheslack-Postava

unread,
May 28, 2016, 6:43:03 PM5/28/16
to Confluent Platform
Saravanan,

Yes, this is possible -- a sink connector could read data and write it to a new Kafka topic (and do any of the transformations you mentioned).

-Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/782a898b-e89e-4531-b70d-5708b89bc657%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
Thanks,
Ewen

Saravanan Tirugnanum

unread,
May 31, 2016, 10:17:10 PM5/31/16
to Confluent Platform
Thank you Ewen!! Will revert back with the results later..

Regards
Saravanan


On Saturday, May 28, 2016 at 5:43:03 PM UTC-5, Ewen Cheslack-Postava wrote:
Saravanan,

Yes, this is possible -- a sink connector could read data and write it to a new Kafka topic (and do any of the transformations you mentioned).

-Ewen
On Fri, May 27, 2016 at 7:38 AM, Saravanan Tirugnanum <vtsa...@gmail.com> wrote:
I have to transport messages between two Kafka Topics and do some transformation before publishing. Understand Kafka Streams will fit this requirement but we cannot migrate to 0.10 Kafka Cluster in near future.. So , my thoughts are to use Kafka Connect Sink Job to move messages between Kafka topics.

We have to build multiple connectors with different sources with Kafka also being one of the source before publishing to final Kafka topic for further processing . 
So , Can Connect  be an option ..

Is this a good approach . Any thoughts..

Regards
Saravanan

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.



--
Thanks,
Ewen

Saravanan Tirugnanum

unread,
Jun 5, 2016, 1:49:25 AM6/5/16
to Confluent Platform
Hi Ewen

Have couple of clarifications on this same thread..

1. From Sink Connector , does json converter\deserializers converts any json messages ( as i would be reading from different source topics) to Connect Schema that can be accessed using SinkRecord.. 

2. Also some of source messages are encrypted , so i would need to apply decryption before converting to Connect Schema.. Should we do this decryption inside the json converters.

Sorry if these are already answered , so you can point me directly, Appreciate if you can respond on this anyways.

Regards
Saravanan

Saravanan Tirugnanum

unread,
Jun 6, 2016, 10:32:24 AM6/6/16
to Confluent Platform
Along with this , i have another challenge ahead in-terms of compatibility as my source topics are running in 0.8 cluster.. Is there any way to use kafka connect sink jobs..
Or is upgrade to latest version the only option..

Regards
Saravanan

Ewen Cheslack-Postava

unread,
Jun 7, 2016, 1:40:32 AM6/7/16
to confluent...@googlegroups.com
Responses inline.

On Mon, Jun 6, 2016 at 7:32 AM Saravanan Tirugnanum <vtsa...@gmail.com> wrote:
Along with this , i have another challenge ahead in-terms of compatibility as my source topics are running in 0.8 cluster.. Is there any way to use kafka connect sink jobs..
Or is upgrade to latest version the only option..

Connect is only 0.9+. It requires consumer and group membership functionality which was not implemented until 0.9.
 

Regards
Saravanan


On Sunday, June 5, 2016 at 12:49:25 AM UTC-5, Saravanan Tirugnanum wrote:
Hi Ewen

Have couple of clarifications on this same thread..

1. From Sink Connector , does json converter\deserializers converts any json messages ( as i would be reading from different source topics) to Connect Schema that can be accessed using SinkRecord.. 

Yes, the converter will deserialize and convert JSON messages into Connect data APIs.
 

2. Also some of source messages are encrypted , so i would need to apply decryption before converting to Connect Schema.. Should we do this decryption inside the json converters.

You'd have to do this as part of the converter as it receives the raw bytes from Kafka. There's no other pluggable component that sees the data before this step.
 

Sorry if these are already answered , so you can point me directly, Appreciate if you can respond on this anyways.

Regards
Saravanan

On Tuesday, May 31, 2016 at 9:17:10 PM UTC-5, Saravanan Tirugnanum wrote:
Thank you Ewen!! Will revert back with the results later..

Regards
Saravanan

On Saturday, May 28, 2016 at 5:43:03 PM UTC-5, Ewen Cheslack-Postava wrote:
Saravanan,

Yes, this is possible -- a sink connector could read data and write it to a new Kafka topic (and do any of the transformations you mentioned).

-Ewen
On Fri, May 27, 2016 at 7:38 AM, Saravanan Tirugnanum <vtsa...@gmail.com> wrote:
I have to transport messages between two Kafka Topics and do some transformation before publishing. Understand Kafka Streams will fit this requirement but we cannot migrate to 0.10 Kafka Cluster in near future.. So , my thoughts are to use Kafka Connect Sink Job to move messages between Kafka topics.

We have to build multiple connectors with different sources with Kafka also being one of the source before publishing to final Kafka topic for further processing . 
So , Can Connect  be an option ..

Is this a good approach . Any thoughts..

Regards
Saravanan

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.



--
Thanks,
Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

Saravanan Tirugnanum

unread,
Jun 7, 2016, 11:31:54 AM6/7/16
to Confluent Platform
Thanks a lot for the clarifications..Just one last ..

Am trying to create a producer record from Sink Record and wondering if i can create a SourceRecord instead so that i can leverage the existing producer functions already available in WorkerSourceTask implementation.

Simply put , I wanted to see if we can convert from Sink Record to Source Record and doing some transformation in between - Can we leverage existing Source Tasks API..
Sorry if am complicating things here :)

Regards
Saravanan
Responses inline.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.



--
Thanks,
Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

Ewen Cheslack-Postava

unread,
Jun 7, 2016, 7:15:38 PM6/7/16
to Confluent Platform
On Tue, Jun 7, 2016 at 8:31 AM, Saravanan Tirugnanum <vtsa...@gmail.com> wrote:
Thanks a lot for the clarifications..Just one last ..

Am trying to create a producer record from Sink Record and wondering if i can create a SourceRecord instead so that i can leverage the existing producer functions already available in WorkerSourceTask implementation.

Simply put , I wanted to see if we can convert from Sink Record to Source Record and doing some transformation in between - Can we leverage existing Source Tasks API..
Sorry if am complicating things here :)


Not sure I understand. A SourceRecord is basically a superset of ProducerRecord, but wouldn't make sense to a Producer. But creating both SourceRecord and ProducerRecord basically amount to copying fields, so both are easy. You won't be able to leverage any of the implementation of SourceTasks in your SinkTask as none of that machinery is created for SinkTasks. I also wouldn't recommend trying to hook into the private APIs as they are not stable and you'd need to modify the implementation with every release of Kafka.

-Ewen
 
Regards
Saravanan
Responses inline.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.



--
Thanks,
Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Thanks,
Ewen
Reply all
Reply to author
Forward
0 new messages