Confluent Control Center - HA

[singh.sh...@gmail.com]

Hello 

- I'm looking for best practices wrt Control Center HA

i.e. 

if we have 2 3C in the same DC, one of them being the primary & the other is secondary..

what needs to be done to enable HA, & failover from primary to secondary 3C

also, if there is documentation on this, pls provide the link.

  

[singh.sh...@gmail.com]

from what i understand, Control Center stores its data in 

a) Kafka topics (_confluent-monitoring, _confluent-metrics, _confluent-command & others) ..

b) RocksDB (i.e. Kafka streams state store)  

for 2 CC instances on the same cluster, the CC would point to the same Kafka topics .. so if primary CC goes down, the secondary CC would still be able to come up, since it is pointing to the same Kafka Topics

wrt the RocksDB - in my case, they are on respective Nodes/server .. so I'll need to sync up the RocksDB on both the Nodes or have the RocksDB data folder on shared drive ?  

 

would that help ensuring HA ?

Can someone pls. help confirm my understanding ? 

[Gwen Shapira]

Here's how I'd do Control Center HA:

1. Get one Control Center working :)
2. On another node, install another Control Center. Take a look at control-center.properties and find "confluent.controlcenter.id" configuration. Change it to a different number. This will force creation of new topics and new RocksDB stores.

3. Put loadbalancer in front, so you can automatically use either UI.

This is fully supported by Confluent and we have customers who implemented it.

Gwen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/70108a8c-b8f3-4355-bdc6-02166d66d0a8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[singh.sh...@gmail.com]

Thanks, 

from what i understand, the topics _confluent-metrics, _confluent-monitoring, _confluent-command will be shared between the two 3C instances ..

while the topics specific to aggregated data (eg. metrics for last 15 mins, 1 Week, 1 Month etc) will be dependant on the confluent.controlcenter.id .. 

However, for argument's sake :) .. if the same controlcenter.id is given to both the instances - would there be an issue ? 

since the base topics ( _confluent-metrics, _confluent-monitoring, _confluent-command) are common between the two 3C instances..

another basic question ->

from what i understand - the RocksDb location is specified in location -> confluent.controlcenter.data.dir ? 

On Monday, October 30, 2017 at 5:57:57 PM UTC-7, Gwen Shapira wrote:

Here's how I'd do Control Center HA:

1. Get one Control Center working :)
2. On another node, install another Control Center. Take a look at control-center.properties and find "confluent.controlcenter.id" configuration. Change it to a different number. This will force creation of new topics and new RocksDB stores.

3. Put loadbalancer in front, so you can automatically use either UI.

This is fully supported by Confluent and we have customers who implemented it.

Gwen

On Mon, Oct 30, 2017 at 5:04 PM <singh.sh...@gmail.com> wrote:

from what i understand, Control Center stores its data in 

a) Kafka topics (_confluent-monitoring, _confluent-metrics, _confluent-command & others) ..

b) RocksDB (i.e. Kafka streams state store)  

for 2 CC instances on the same cluster, the CC would point to the same Kafka topics .. so if primary CC goes down, the secondary CC would still be able to come up, since it is pointing to the same Kafka Topics

wrt the RocksDB - in my case, they are on respective Nodes/server .. so I'll need to sync up the RocksDB on both the Nodes or have the RocksDB data folder on shared drive ?  

 

would that help ensuring HA ?

Can someone pls. help confirm my understanding ? 

On Wednesday, October 25, 2017 at 10:24:28 AM UTC-7, singh.sh...@gmail.com wrote:

Hello 

- I'm looking for best practices wrt Control Center HA

i.e. 

if we have 2 3C in the same DC, one of them being the primary & the other is secondary..

what needs to be done to enable HA, & failover from primary to secondary 3C

also, if there is documentation on this, pls provide the link.

  

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

[Gwen Shapira]

1. You are right. The inputs are shared and the intermediate state is not.
2. I have no idea :) We are not testing Control Center this way. It may work... streams is a distributed system after all. Let us know :)

3. Yes. But note that it still needs to have write access to /tmp (RocksDB places few dynamic libraries there for some reason).

Gwen

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/70108a8c-b8f3-4355-bdc6-02166d66d0a8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/26bb18fc-79a7-4aa5-bcd9-8b8ee14c7798%40googlegroups.com.

[xav...@confluent.io]

However, for argument's sake :) .. if the same controlcenter.id is given to both the instances - would there be an issue ? 

since the base topics ( _confluent-metrics, _confluent-monitoring, _confluent-command) are common between the two 3C instances..

Control Center will not work correctly if you give the same ID to different instances. Some parts of Control Center currently rely on all stream threads running on the same host.

[singh.sh...@gmail.com]

from what i understand - 

the Control Center actually writes aggregated data to the topics (eg. _confluent-controlcenter-3-2-2-<control-center-id>-group-aggregate-topic-FIFTEEN_SECONDS,  and that data is used to display the Data Stream UI. 

So, if one instance of the 3C is down for a length of period, the writes(or publishing) to the aggregate topic specific to that 3C will not happen.

Since I have 2 instances of 3C installed, once the (down)3C is brought up, the Control Center UI (Streams Monitoring UI) on the two instances  will show different progress.. till the 3C lagging behind catches up.

Can someone pls confirm my understanding ?

thanks

[Xavier Léauté]

Your understanding is correct. If one instance (with a different ID than the other one) is down for a while it could be behind for some time.

However Control Center will not attempt to catch up all the data. For instance, it will not catch up more than 15min worth of data when it comes back online to avoid staying behind too long.

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/6iCS1PBU-os/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/5a1a7536-4f4b-4f69-9264-75ac03b04ad8%40googlegroups.com.

[singh.sh...@gmail.com]

got it, thanks !

On Friday, November 10, 2017 at 9:39:36 PM UTC-8, Xavier Léauté wrote:

Your understanding is correct. If one instance (with a different ID than the other one) is down for a while it could be behind for some time.

However Control Center will not attempt to catch up all the data. For instance, it will not catch up more than 15min worth of data when it comes back online to avoid staying behind too long.

On Thu, Nov 9, 2017 at 2:50 PM <singh.sh...@gmail.com> wrote:

from what i understand - 

the Control Center actually writes aggregated data to the topics (eg. _confluent-controlcenter-3-2-2-<control-center-id>-group-aggregate-topic-FIFTEEN_SECONDS,  and that data is used to display the Data Stream UI. 

So, if one instance of the 3C is down for a length of period, the writes(or publishing) to the aggregate topic specific to that 3C will not happen.

Since I have 2 instances of 3C installed, once the (down)3C is brought up, the Control Center UI (Streams Monitoring UI) on the two instances  will show different progress.. till the 3C lagging behind catches up.

Can someone pls confirm my understanding ?

thanks

On Wednesday, November 1, 2017 at 1:37:40 PM UTC-7, xav...@confluent.io wrote:

However, for argument's sake :) .. if the same controlcenter.id is given to both the instances - would there be an issue ? 

since the base topics ( _confluent-metrics, _confluent-monitoring, _confluent-command) are common between the two 3C instances..

Control Center will not work correctly if you give the same ID to different instances. Some parts of Control Center currently rely on all stream threads running on the same host.

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/6iCS1PBU-os/unsubscribe.

To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.