Rabbit MQ - Architecture

[sarju Garg]

Hi,

I am new to Rabbit MQ and trying to use this in one mobile network based messaging product.

I want to build HA architecture for about 30K TPS solution with message size of 1k. 

I have read that a single node can support upto 50K TPS . Can someone help with server configuration for same.

I was thinking of having the following architecture

1. Warren based HA architecture with Pacemaker for Virtual IP.

2. Mirrored Queue. I am planning to use normal mirror queue instead of quorum as this is only 2 node cluster (warren active standby configuration)

3. I will define multiple vhosts so that some queue will reside on one server and some queue will reside on other server so that incoming load from the producer / consumer is distributed across both nodes.

Is this architecture ok with 30 K TPS with about 20 core/64 GB kind of configuration.

Can someone help and suggest if this is ok.

Regards

Sarju

[Michal Kuratczyk]

Hi,

We have published a lot of test results that will give you an idea of what you may need:

https://blog.rabbitmq.com/categories/performance/

https://www.youtube.com/watch?v=T8PspEW_8J0

We also provide an app to easily simulate all kinds of traffic so you can validate your architecture/infrastructure before writing the applications or having real traffic: https://rabbitmq.github.io/rabbitmq-perf-test/stable/htmlsingle/

Queues are spread between nodes regardless of the vhost - you can have just 1 vhost with queue leaders on different nodes.

You should use quorum queues. Mirrored queues will not be supported in the future. By using mirrored queues, you are designing a system that is "legacy on arrival".

And yes, the cluster should have 3 nodes for this.

Best,

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/384ef1f9-220f-441c-9c29-494376c599fan%40googlegroups.com.

--

Michał

RabbitMQ team

[sarju Garg]

Thanks Michael,

I have read the article on performance and it talks about the direct Q while benchmarking the performance. 

I still have few more doubts when it comes to Rabbit MQ internal architecture especially on Topic/sharding concepts.

For example, the producer connects with one node. So the TCP connection of the producer is with 1 node only. Same is applicable for consumer.

So if there are 5 nodes, and one queue and 5 producer/consumer each connected with one node, still the queue will be on one server. Any TCP traffic coming to node will be routed to the node hosting the queue. So there will be lots of TCP traffic that will be flowing between the nodes to serve producer/consumer. Even if the sharding concept is used, the queue will be made on all the nodes but I assume this TCP traffic would be reduced by 20% assuming 20% of traffic is available on the queue where producer/consumer is also connected. 

How can this internal TCP traffic can be reduced? Having multiple queue is a solution, but we are planning to create queues based on the traffic and we have primarily 3 kind of traffic.

Regards

Sarju

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CAA81d0scmRV%3DvOZ3%3DwDZfsLK%3Dt78M-0UxOOE2yMi_xkhLy2rFA%40mail.gmail.com.

[Michal Kuratczyk]

Hi,

Have you run any tests that would confirm that this is a problem? We have an app that allows to quickly run workload simulations: https://github.com/rabbitmq/rabbitmq-perf-test. It might be that you are trying to optimize something that doesn't need to be optimized. Sure, there is some overhead of being connected to the node that doesn't run the queue you produce to/consume from but plenty of users use RabbitMQ like that - clients connect to random nodes and queues are randomly distributed across nodes (and occasionally rebalanced).

Having a single queue is much more of a scaling problem - a queue can only use one CPU. Also, if you are thinking of using classic mirrored queues, please use quorum queues instead.

If your goal is to limit the inter-node network traffic to save the cost of the public cloud infrastructure, then you might be interested in https://www.rabbitmq.com/clustering-compression.html.

Best,

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CAHt0Csm0rvb0MiWZjmTFUv6erWmjAkAcMoYCXiu6j_2B5q2kdw%40mail.gmail.com.

--

Michał

RabbitMQ team

[sarju Garg]

Hi Michal,

Appreciate that you responded on same.

I have not run any test but still working on the architecture bit and figuring out how to design a high scalable/available messaging system. I am still locked in this thought on how to reduce inter server TCP load. 

Do we have any study done to see what performance metrices is achieved in this configuration

I think single queue problem can be solved thru the sharding where I believe queue are present on different nodes and each client connect to different nodes. Is this understanding correct.

I saw the article but this looks available in commercial version. is this feature planned in the open source rabbitMQ  version

Regards

Sarju

[Michal Kuratczyk]

Just use perf-test to run the experiments you need. Don't hypothesise about the performance problems if you can easily measure the actual impact of different configurations.

Inter-node traffic compression is a commercial feature. If you deploy to a public cloud, you can save a lot on networking costs. We have a calculator to help you estimate the savings: https://www.rabbitmq.com/tanzu/#calculator

Best,

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/0cef6b3b-a937-4cf4-a152-9032fd1d609bn%40googlegroups.com.

--

Michał

RabbitMQ team

[Carlos Alberto Balseiro Mayi]

You should use quorum queues. Mirrored queues will not be supported in the future. By using mirrored queues, you are designing a system that is "legacy on arrival".

And yes, the cluster should have 3 nodes for this.

Is there any suggested procedure for scenarios where currently two nodes with mirrored queues are running in with very low latency between them, and a third node will necesarily be deployed in a remote location with much higher latency?

[Michal Kuratczyk]

Hi,

This kind of topology is asking for trouble. From https://www.rabbitmq.com/distributed.html#clustering:

The design of clustering assumes that network connections are reasonably reliable and provides a LAN-like latency.

As long as "much higher latency" is still very low latency and the reliability is very high then it could probably work but I definitely wouldn't assume that without thorough testing. If you want geographically distributed nodes, use an AZ setup similar to public clouds (3 AZs with very low latency and very high reliability).

Best,

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/0b287d5f-1e9b-4273-8c66-e1ebc7eb60e3n%40googlegroups.com.

--

Michał

RabbitMQ team