Kafka connect standalone vs distributed

[Skrzypek, Jonathan]

Hi,

 

Reading through the kafka connect documentations I'm wondering what's the difference in terms of consumer groups between standalone and distributed.

It seems that to run multiple consuming processes on different hosts the distributed setup is recommended.

But it looks like the standalone mode creates a kafka consumer group anyways, so if you run multiple standalone instances, doesn't it allow you to scale in the same way ?

Each standalone process will consume from one or more partitions right ?

 

Also, in the distributed mode, can multiple workers or tasks consume from the same partition ?

 

Thanks

 

 

[Dustin Cote]

Hi Jonathan,

The biggest difference between standalone and distributed mode is the fact that the workers know about each others existence in distributed mode, allowing you to have some fault tolerance and coordination between workers:

http://docs.confluent.io/3.0.0/connect/userguide.html#distributed-mode

If you run multiple standalone instances, you would have to coordinate them yourself and you would have additional resource overhead to manage as well.

In terms of what can consume from the same partition, it's more about the connectors than the workers themselves.  The logical separation looks like this:

http://docs.confluent.io/3.0.0/connect/userguide.html#connectors-tasks-and-workers

You can then imagine a second connector in that diagram as an independent entity just like an independent consumer group would work.  The different logical breakdowns for connect are described here:

http://docs.confluent.io/3.0.0/connect/design.html#architecture

-Dustin

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/F6A8DEE6B40A30419B89834C7CFF43F4032DCD00%40gsdgeup01env2.firmwide.corp.gs.com.
For more options, visit https://groups.google.com/d/optout.

--

Dustin Cote

confluent.io

[Skrzypek, Jonathan]

Thanks !

In my case I have a single simple task that I want to parallelize leveraging partitions, they don’t have to share state or do anything on rebalance or failover, so I guess running as many standalone instances as partitions does the job.

The underlying consumer api with the consumer group should coordinate which consumers gets data from which partition.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CACOCneZD2B0iAKhXXq5y1QDLkH0kSHuK_LjM1Qd6FginaOCUdQ%40mail.gmail.com.