RabbitMA HA across data centers?

[Jim Archer]

Hi All...

I have been asked to configure RabbitMQ HA for our new system. We have a cluster of public facing web servers which reside in two different data centers. I planned to use RabbitMQ to allow each server to send log and other status messages relevant to our application to some backend servers. Some folks here would prefer that Rabbit run redundantly. As I was reviewing the docs (https://www.rabbitmq.com/ha.html), I ran across a paragraph I don't fully understand, although it looks like bad news for me:

This solution requires a RabbitMQ cluster, which means that it will not cope seamlessly with network partitions within the cluster and, for that reason, is not recommended for use across a WAN (though of course, clients can still connect from as near and as far as needed).

 

Does this imply that Rabbit can't be used redundantly across data centers at all? Or perhaps I can not use a network partition? 

[Tony]

RabbitMQ clusters do not tolerate network outages very well.  As such, LAN<->WAN<->LAN clusters is not recommended across your entire environment.  If you want to replicate a portion of your queues to the second data center take a look at Shovel and/or federation.  See the bottom of this page for links...

https://www.rabbitmq.com/reliability.html

[Michael Klishin]

On 14 Oct 2015 at 21:34:59, Jim Archer (bostc...@gmail.com) wrote:
> Does this imply that Rabbit can't be used redundantly across
> data centers at all? Or perhaps I can not use a network partition?

It should not be used across WAN links. It technically can be.

The primary reason is that it’s not DC/rack aware, so data locality
will be horrible. 
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Laing, Michael]

In your situation, we run clusters in our datacenters (LAN connections) and shovels/federation between datacenters (WAN connections). No system failures in non-stop production for several years, although every single component has failed and/or been replaced during that time.

ml

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send an email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jim Archer]

Okay thanks everyone, I'll check shovel and federation out. 

If I decided, temporarily, to run just one instance of RabbitMQ at one of the DCs, and had all the servers connect to it, how well would that work? If a queue is lost will it reconnect, or should I write code to watch for that?

 

[Michael Klishin]

On 14 Oct 2015 at 22:40:09, Jim Archer (bostc...@gmail.com) wrote:
> If I decided, temporarily, to run just one instance of RabbitMQ
> at one of the DCs, and had all the servers connect to it, how well
> would that work? If a queue is lost will it reconnect, or should
> I write code to watch for that?

Sorry, I don’t understand the idea.

What exactly do you mean by “queue lost”? A node going down? Several client
libraries offer automatic connection recovery, for example. So do Shovel
and Federation. 

[Jim Archer]

I meant if there is only one instance of RabbitMQ and a client lost connectivity briefly, if the queue would reconnect. My application is a Spring Boot using the RabbitMQ Spring library.

But I have been reading the Shovel and Federation docs as you all suggested and it seems sensible enough. I'm not sure I yet understand the difference between the two. From looking at Federated docs, it seems that one of the simple examples would work for me. The first example shows two clusters (in my case, each "cluster" would be one instance only) connected together bidirectionally. I have two data centers, each will have one instance of RabbitMQ and I could put Federation on each. 

Alternatively, I maybe can use can use the fanout example. I say maybe because I'll have producers in both data centers. My architecture basically just has 2 or more producers broadcasting messages to a logs queue, and a bunch of clients (some java apps, some web browsers) consuming that data.

I'm not sure how Shovel fits in.

[Gary Russell]

Regarding Spring connection management:

If you have async Spring consumers (message listener containers), the connection will be re-established when the broker is available (according to the container's recoveryInterval (5 seconds by default).

On the producer side, you can add a RetryTemplate (configured with whatever retry policy you want) to the RabbitTemplate.

Otherwise, message publishing will fail until the broker is available again.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.

[Jim Archer]

Thank you Gary, I will definitely do that.

[Michael Klishin]

On 14 Oct 2015 at 23:20:19, Jim Archer (bostc...@gmail.com) wrote:
> I meant if there is only one instance of RabbitMQ and a client
> lost connectivity briefly, if the queue would reconnect. My
> application is a Spring Boot using the RabbitMQ Spring library.

Java client supports automatic connection recovery (this is opt-in),
 so does Spring AMQP.