Kafka Connect in Distributed mode

[Pradeep Sadashivamurthy]

Trying to understand Kafka connector to set it up. I would like to develop and deploy Kafka connector in Distributed Mode on AWS EC2 instance. But the Kafka server is running on a on-premise Linux box in my office.

1. Is this set up possible? 

2. How to deploy the Kafka connector jar file on EC2? 

3. Is it possible to scale the Kafka connector on EC2 instance.

4. Is it possible to run the Kafka connector on multiple EC2 instances?

Many Thanks

Pradeep

[Randall Hauch]

Hi, Pradeep.

Each Kafka Connect distributed worker that is to form a single cluster must have the same "group.id", and the "boostrap.servers" must point to the same Kafka broker cluster. Generally all of the other configuration settings will also be the same.

1. It is as long as your EC2 instance has network connectivity to your brokers.

2. Please see Confluent's documentation at http://docs.confluent.io/3.2.1/connect/index.html, including the section about installing connector plugins. Basically, the connector's JARs must be on Kafka Connect's classpath.

3. You can add any number of distributed workers to your Kafka Connect cluster, and Kafka Connect will distribute the tasks of your deployed connectors. You will only need as many workers as the total number of tasks, and each connector will decide how many tasks it can use. Again, there's more on this in the documentation.

4. It doesn't matter whether your Kafka Connect workers are run on one EC2 instance or multiple. They will discover each other as long as each have network connectivity to each other and to the brokers.

Hope that helps,

Randall

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/36545530-9a22-4913-b0d6-209e108779ae%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Pradeep Sadashivamurthy]

Hi Randall

Thanks a lot for the quick response. Please correct me if I’m wrong.

 So to deploy the connectors, the connector’s JARs needs to be in Kafka Server’s (on on-premise local linux box) classpath. So how can I set classpath to Kafka Server if the connector jar is on EC2?

If the jar is on Kafka Server class path then the connector will be on both  on on-premise local linux box and EC2?

Many Thanks
Pradeep Sadashivamurthy

You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/ivsN9bVdocM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CALYgK0EBbn6BTa3t8%2BfGaL2SPce0Xj-ejzuxamCKOTAyYAUPkw%40mail.gmail.com.

[Randall Hauch]

The connector's JARs do not have to be on the classpath of the *broker*. They *do* have to be on the classpath of the Connect, but the Connect scripts (e.g., "connect-distributed.sh" and "connect-standalone.sh") allow you to specify the directories where the plugins are installed via the CLASSPATH environment variable. Please see http://docs.confluent.io/3.2.1/connect/userguide.html#installing-connector-plugins for details.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/36545530-9a22-4913-b0d6-209e108779ae%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/ivsN9bVdocM/unsubscribe.

To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CALYgK0EBbn6BTa3t8%2BfGaL2SPce0Xj-ejzuxamCKOTAyYAUPkw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/0A652086-2CAA-47BD-9FC8-C226DFBCEDC4%40gmail.com.

[Pradeep Sadashivamurthy]

Kafka Connect is included in Kafka broker (confluent-3.2.0) package. Here is my scenario, I have a Kafka broker installed on machine A and I need to set up the Kafka-Connect connector from another machine B. Assume that all the Kafka broker and zookeeper ports on machine A are open to machine B.

So should I install confluent-3.0.1 on machine B (since Kafka Connect is part of the Kafka Broker package) and configure the custom Kafka-Connect connector jar by setting the class-path? What are the setting should I be looking at to make this setup work? 

To unsubscribe from this group and all its topics, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CALYgK0ExjfSYdagpN012-oHaTfGc3CGFHD0GTS7di7b-n0bRyQ%40mail.gmail.com.

[Randall Hauch]

The Confluent Platform does have everything in it, since that simplifies everything. It does not hurt to have the same installation everywhere, and in fact it is usually much easier.

So, yes, go ahead and install the same Confluent Platform all machine A and B, and make sure that your custom connector JARs are installed on at least machine B, and set the classpath as described at http://docs.confluent.io/3.2.1/connect/userguide.html#connect-installing-connector-plugins when running Kafka Connect worker(s).

You can run a cluster of multiple Kafka Connect workers on machine B, or run the workers on multiple machines as desired (e.g., B1, B2, B3,...). If running multiple workers as a cluster, make sure that the configurations for all workers in the cluster are the same (as I mentioned earlier).

To unsubscribe from this group and all its topics, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CALYgK0EBbn6BTa3t8%2BfGaL2SPce0Xj-ejzuxamCKOTAyYAUPkw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/0A652086-2CAA-47BD-9FC8-C226DFBCEDC4%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/ivsN9bVdocM/unsubscribe.

To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CALYgK0ExjfSYdagpN012-oHaTfGc3CGFHD0GTS7di7b-n0bRyQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/C8B68460-3809-419E-A686-CB10831E6004%40gmail.com.

[Pradeep Sadashivamurthy]

Thanks a lot Randall. I really appreciate you help.

Many Thanks

Pradeep

Many Thanks

Pradeep Sadashivamurthy

Ph - 07729776432

To unsubscribe from this group and all its topics, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CALYgK0ExjfSYdagpN012-oHaTfGc3CGFHD0GTS7di7b-n0bRyQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/C8B68460-3809-419E-A686-CB10831E6004%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/ivsN9bVdocM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CALYgK0FhQjFrSbqC%3DpR5Zks4eW29CjNtf077_J5aPZsfhk%2Bi%2Bw%40mail.gmail.com.