RabbitMQ 3.6 does not flush memory when reached high watermark

[Subash Chaturanga]

Hi team,
I have a rabbitmq cluster of 3 nodes running with default configs except memory high water mark is set to 1G(my RAM is 8G). I am trying to see how rabbitmq can withstand bad consumers.

Issue is:
Persistent messages publish to a direct exchange around 3000 tps, and the bounded queue does not have a consumer. And as expected it reached memory watermark ~1G and then publisher get blocked, which is good. But I would expect at some point RabbitMQ will flush the RAM and let publisher hit more messages. But it keeps blocking connections forever and in queue details I can see all messages are in memory as well as in disk. Is there a way to let RabbitMQ to ask to clear memory. I know I can either purge or hook up the consumer back to drain them. But I am looking at a automated way. Thought vm_memory_high_watermark_paging_ratio does that, which seems does not.

This is RabbitMQ 3.6.6 and client is Java.

[Michael Klishin]

> I can see all messages are in memory as well as in disk

You publish messages as transient (it is the default in most if not all clients),

thus instructing RabbitMQ to keep them in RAM as much as possible.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send an email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Subash Chaturanga]

Hi MK,

Thank you very much for the prompt response. 

I am using a simple java code, where BasicProperties#deliveryMode is set to 2. Which is persistent. Following is the code.

channel.basicPublish(exName, rk,
                 new AMQP.BasicProperties.Builder()
                .contentType("text/plain")
                .deliveryMode(2)

                .timestamp(new Date())

                .messageId(id)
                .build()), body);

[Michael Klishin]

If you want RabbitMQ to keep as few messages as possible in RAM,

use lazy queues:

http://rabbitmq.com/lazy-queues.html

With 3.6.7 previews we have test environments that enqueue 10s of millions of messages (or gigabytes)

without consumers present.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

Oh, and make sure your queue is durable: persistent messages in non-durable

queues are treated as transient (because their queue will be deleted on server restart).

Consider posting your entire code snippet when asking questions in the future.

To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Subash Chaturanga]

Hi all,

Queue is durable and I double checked it from the UI under queue features durable=true.

About lazy queues, exactly. I did a test some time back and as you correctly mentioned memory is not growing up at all. So that I kept as a work around, reason is when consumer acknowledges every message, with lazy queues I was able to maintain a 1600 tps on end to end, and with durable non lazy queues, I am able to get 3000 tps. (FYI, Without acks I was able to achieve >9000 tps on 4 core 8G boxes in each scenario). And this drop with lazy queues seems obvious because of disk reads.(my consumer utilization is 100% in each case where I kept 5 consumers for the same queue using 5 connections)

So this is the reason I raised this question, where if RabbitMQ can flushes its messages in memory when 

Memory alarm kicked in or on its way this can be solved and we can size broker boxes with fair amount of memory like 8G and also have a good TTL for messages.

Sorry about not putting the whole code. The code is not in one place/class but abstracted and wrapped in different layers. So here is what I extracted to read easily.

I am using 3.6.5 version with Erlang R16B03

Policy for the queue.

queue-master-locator:  min-masters

ha-mode:            all

ha-sync-mode: automatic

 

Queue declare and bind before publish.

Connection connection = newConnection(n, fac);  
Channel channel = connection.createChannel();
channel.queueDeclare(queueName, true, false, false, null);
channel.queueBind(queueName, exchange, routingKey);

 

public static Connection newConnection(Broker n, ConnectionFactory factory) throws Exception {
    factory.setConnectionTimeout(MQ.getConfig().getConnectionTcpTimeout());
    factory.setUsername(n.getUsername());
    factory.setPassword(n.getPassword());
    factory.setVirtualHost(n.getVhost());
    factory.setHost(n.getHost());
   factory.setPort(n.getPort());
    addShutdownExecutor(factory);
    return factory.newConnection();
}

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/BKuEDVTrp-Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Subash Chaturanga

Blog -  http://subashsdm.blogspot.com/

Twitter - http://twitter.com/subash89

 

[dfed...@pivotal.io]

Hi,

RabbitMQ has several levels of caching before it tries to actually write messages to disk. And it won't read them from disk if they are cached in memory.

So lazy queues are the way to flush messages from memory to disk. And they will be written and read from disk only if consumers cannot keep up with producers.

Publisher confirms have different purpose. They are required to make sure messages are not lost due to network/system errors. If you can afford to loose messages - you can disable them, if you cannot - you have to use them and have your benchmarks with confirms.

In case you cannot afford to loose messages, you supposed to enable lazy queues, so enqueued messages will be preserved if RabbitMQ server fails. In that case your performance is effectively limited by disk write speed, as message will be confirmed as soon as it's written to disk.

If you have end-to-end confirmations and/or can afford to loose messages during network and server failures - you can disable confirms, and use lazy queues as a buffer to avoid memory alarms.

[Michael Klishin]

RabbitMQ's core protocol has a feature which lets clients decide whether a message should
be stored in RAM or on disk. It's a very unfortunate feature in terms of operations so
RabbitMQ tries to find a balance: flush messages out to disk when it's under memory pressure
and keep things in RAM while it can. The details of this are pretty involved and
this can be to some extent controlled via `rabbit.vm_memory_high_watermark_paging_ratio`
(by *reducing* it, not increasing, and not setting it too low; the default is 0.5), see
http://www.rabbitmq.com/memory.html.

Lazy queues is a way to tell RabbitMQ "screw that, move things to disk as early as possible".

> > channel.queueDeclare(queueName, *true*, *false*, *false*, *null*);
> > channel.queueBind(queueName, exchange, routingKey);
> >
> >
> >
> > *public static *Connection newConnection(Broker n, ConnectionFactory
> > factory) *throws *Exception {
> > factory.setConnectionTimeout(MQ.*getConfig*
> > ().getConnectionTcpTimeout());

> > factory.setUsername(n.getUsername());
> > factory.setPassword(n.getPassword());
> > factory.setVirtualHost(n.getVhost());
> > factory.setHost(n.getHost());
> > factory.setPort(n.getPort());

> > *addShutdownExecutor*(factory);
> > *return *factory.newConnection();

> >> For more options, visit https://groups.google.com/d/optout.
> >>
> > --
> > Subash Chaturanga
> >
> > Blog - http://subashsdm.blogspot.com/
> > Twitter - http://twitter.com/subash89
> >
> >
>

> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users"
> group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

> To post to this group, send an email to rabbitm...@googlegroups.com.

[Subash Chaturanga]

Hi MK,

Previously I meant acks enabled as consumer side ack to broker. From publisher side our use case is auto ack true.

Can you please elaborate this for me.

" RabbitMQ tries to find a balance: flush messages out to disk when it's under memory pressure

and keep things in RAM while it can "

As I understood, rabbit.vm_memory_high_watermark_paging_ratio controls how frequently you want to page out messages to disk. But even then, messages can still be in memory. So when you say keeps things in RAM while it can, does it mean it keep them until queue purges, or until a consumer comes to deliver them ? Is it correct to understand that by design RabbitMQ does not clear the RAM in this situation. If so would you like to share the reasons behind that.

To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

To post to this group, send an email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

[Michael Klishin]

I shared the reason behind this in my previous email: one of the protocols (and the first one RabbitMQ implemented) allows publishers to publish messages as transient, which instructs RabbitMQ to keep messages in memory. Another obvious reason is that then they can be delivered with lower latency to consumers, compared to a case where they are only kept on disk.

Messages are marked as deleted when they are acknowledged by consumers. In automatic acknowledgement mode this means as soon as they were written down the

socket without errors.

There is a separate thread that was started (I'm guessing by a colleague of yours) that goes into a bit more detail about when messages are persisted in general.

To post to this group, send email to rabbitm...@googlegroups.com.