Re: [rabbitmq-users] Tune Configuration to Handle 60..120 GB/hour

[Michael Klishin]

Lazy queues and a lot of disk space. As soon as one of the mirrors will need

to sync from scratch your network will be maxed out for a very long time, though,

so a much wiser idea is: store the payloads as files in a blob store/service

and pass keys around in messages instead.

On Thu, Jun 28, 2018 at 3:09 PM, <o.kur...@gmail.com> wrote:

Hello dear RabbitMQ Community and Dev Team

I'd appreciate your help with RabbitMQ Server tuning. 

In a nutshell, the question is: how should I alter default configurations in order to accomodate 10 - 20 GB of messages every minute and survive absence of consumption as long as possible?

The main aims are: 

1. Utilize available hardware in the most optimal way to achieve maximum throughput
   
2. Achieve maximum availability, enforcing back-pressure control when necessary

Current custer layout:

2 Nodes, linked with full duplex 10Gbps link (10 Gbps TX/10 Gbps RX). 

Hardware: 

Each Node has:  

    * 128GB RAM 

    * Ubuntu 14 OS 

    * 12 CPU Cores 

    * 1.8 TB SSD RAID 10 (capable ~20 KOps, write rate ~1.7 GB/s)

    * 2X 10Gbps NICs (1 for producers/consumers, 1 for inter-node communication) 

Schema: 

1 Exchange: X, 1 queue: Q. 

We publish messages to X, using routing key. Message gets routed to the Q.

Q is Lazy, Durable, Mirrored.

Messages are persistent.

Workload:

Once a minute every producer publish 1 message to any of 2 nodes (round-robin)

Message size -> Amount of Producers:

50 MB  ->  50 producers

35 MB  ->  50 producers

20 MB  -> 100 producers

10 MB  -> 100 producers

5 MB   -> 500 producers

2.5 MB -> 500 producers

0.5 MB -> 200 producers

Overall: 

11.1 GB/min -> 1500 producers

RabbitMQ Server specs: 

Erlang/OTP 20.3

RabbitMQ Server 3.7.6-1

 

I'll add results of stress tests later on.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ