Paging doesn't clear memory

[Rob A.]

Hi all,

Currently we are facing memory issues in our rabbitmq 3.7.9. single node production environment.

One of our consumers (xosd) is sometimes not able to process messages (fast enough) and it's queue is filling up to 3000-5000k messages (with 1-3 MB message size).

Our rabbitmq node has 20 GB memory, with high_memory_watermark of 0.5 and the default paging threshold of 0.5.

Therefore paging should happen at 5GB memory usage.

As you can see we are loosing a lot of prometheus metrics, as prometheus is not able to connect to rabbitmq prometheus plugin anymore, starting from 5 GB memory usage.

According to the manual[1] paging should stop producers during paging, but it seems also connections are blocked. Is this assumption correct?

Also paging doesn't free enough space.

All messages in "xosd" queue are persistent, the queue is durable, but not in lazy mode (we are thinking about activating it, but I would like to understand the default behavior).

As far as I understand the documentation paging should massively reduce the memory usage, quite fast, as most messages are already persisted.

But I don't see any improvement in the graphs.

Theoretically it should never reach the high watermark.

Also there are no log entries which would show paging was started. Are there no logs, when paging runs?

I only see logs when the high memory watermark is reached.

Thank you and best regards

Roberto

[1] https://www.rabbitmq.com/memory.html#memsup-paging

[Rob A.]

Hi,

Now we faced a serious issue in production again.

In order to prevent these issues we doubled the RAM before. 40 GB RAM, 20 GB high watermark.

But last night our rabbitmq node used more than 25 GB RAM. All messages were persistent and even after they have been removed (TTL = 15 minutes), the RAM usage didn't decrease.

All producers where blocked until we recreated the node.

[Rob A.]

Btw. we are using erlang 21.2.2 with TLS.

Is it possible this bug may cause this problem?

https://bugs.erlang.org/browse/ERL-934

[Luke Bakken]

Hi Rob,

That is a possibility. You should try upgrading to 21.3.8.2

Thanks -

Luke

[Rob A.]

Hi Luke,

We will consider upgrading soon.

Now, I've tried using lazy queues, but it didn't help.

Even if all messages are persisted in lazy mode and I restart the node it used 40 GB of rss RAM instantly.

And if I purge most messages and then restart the node it first allocated 30 GB RAM, then decreased to rss 700 MB [1].

I also captured the RAM usage before switching to lazy mode.

As you can see most RAM is not directly allocated by data, but from erlang [2].

It seems to be a memory leak somehwere, or an error in the garbage collector.

[1]

[2]

{memory,

     [{connection_readers,14584212},

      {connection_writers,599240},

      {connection_channels,2257084},

      {connection_other,20160052},

      {queue_procs,4866176},

      {queue_slave_procs,0},

      {plugins,10757228},

      {other_proc,78505332},

      {metrics,2195012},

      {mgmt_db,12779264},

      {mnesia,527712},

      {other_ets,4157424},

      {binary,24412160720},

      {msg_index,296256},

      {code,23760359},

      {atom,1812721},

      {other_system,40830848},

      {allocated_unused,5273175896},

      {reserved_unallocated,0},

      {strategy,rss},

      {total,

          [{erlang,24630249640},{rss,22018035712},{allocated,29903425536}]}]},

[Luke Bakken]

Hi Rob,

I have a few questions, as I was going to suggest lazy mode:

* Your messages are 1 - 3MiB in size, correct?

* Do you have any custom RabbitMQ configuration in place?

* Can you please share how you are declaring your lazy queues?

The reason I ask is that I have recently tested just this scenario - messages published to a lazy queue - and none were loaded into memory on node restart:

https://groups.google.com/d/msg/rabbitmq-users/-u_D_U9tGJo/jOgsLAJrDAAJ

If you declare a queue as lazy *after* publishing messages to it the lazy setting will have no effect on restart. You must declare the queue as lazy when it is first declared.

Thanks -

Luke

[Rob A.]

Hi Luke,

I have analyzed the messages in detail and during the memory spikes we receive 1-3k messages with 20-30 MB in size.

I've applied a policy to change the queues to lazy mode.

So even if I use a policy they will be in default mode after restart, until all messages have been loaded to RAM?

Last night we processed a batch of big messages in lazy mode.

The broker didn't crash, which is an improvement.

But, we still reached the high watermark:

Garbage Collection seems to work:

Only custom properties are related to SSL(ciphers and authentication) and high watermark of 0.49.

Thanks

Rob

[Rob A.]

Hi again,

We have processed another Batch of 5000 large messages and I tried to capture screenshots from the management ui.

Again the system became very slow and memory increased.

Most messages stuck in this queue, all messages should be persisted an no message in RAM:

But during this time the memory was 12 GB for binaries:

Later on it increased to 12 GB (but all messages in all queues were still persisted):

The binary reference shows 4 GB "other"

[Luke Bakken]

Hi Rob,

Thanks for providing that information, I can try to reproduce what you report in my local environment using PerfTest and large message bodies. For what it's worth, a 20MiB message is pretty dang big so we recommend compressing the message body if possible.

Are these large messages coming from a single publisher or multiple at the same time?

It does appear that the lazy queues are working correctly as it shows 0 messages "in memory". The large amount of memory being used is probably due to the data being read from the socket into memory before being written to disk.

Thanks,

Luke

[Luke Bakken]

Hi again,

I've done some tests using my local environment - RabbitMQ 3.7.15 / Erlang 21.3.8.3 and our PerfTest tool.

In my first test, I'm using these PerfTest arguments, which publish 30MiB persistent messages using four producers. The test-lazy queue has been declared as a lazy queue:

--predeclared --queue test-lazy --flag persistent --pmessages 1024 --producers 4 --consumers 0 --uri amqps://my-rabbitmq-server:5671 --size 31457280

I'm running RMQ on a low-horsepower home server (8GiB RAM, spinning disks, i3-3220 CPU @ 3.30GHz) and am using a 1Gbps ethernet link to my workstation. I'm using TLS and the default memory watermark of 40%.

During the test run, I do see memory reach the high watermark (3GiB) which is to be expected as RabbitMQ can't page messages to disk fast enough to keep up. The node doesn't crash, however, like you see in your tests.

I did another test that adds the --confirm 16 argument to PerfTest. What this does is enable publisher confirms, with the requirement that the backlog of confirmed messages doesn't exceed 16 outstanding confirmations. This functions as a throttle on publishing speed and allows messages to be paged to disk before the high watermark is hit. I don't see memory use exceed 2.3GiB or so during this test run.

It didn't seem to matter if I used Erlang 21.2.2 or 21.3.8.3.

In your environment, I recommend doing the following:

* Compress your message bodies

* Use publisher confirmations in your producers. Don't synchronously wait for a confirmation after publishing a message, but keep an "acceptable number" of published messages awaiting confirmation that, if about to be exceeded, blocks further publishing. Receiving N confirmations means that N more messages can be published, etc. Please note that confirmations can be received for multiple messages as well - https://www.rabbitmq.com/confirms.html#publisher-confirms

Let me know if you have any more questions -

Thanks,

Luke

[Rob A.]

Hi Luke,

Thanks for putting time into the analysis.

Currently using lazy queues our node doesn't crash anymore, but during the spikes with big messages it still reaches the high watermark and becomes very slow for a couple of minutes.

I still don't understand why it reaches the high watermark.

In your test your disk was too slow, this would be an explanation.

But we've already verified this is not the case.

We are using an AWS EBS (gp2) volume which can handle 250 MB/s.

Our peaks are at 160-180 MB/s and our IOP burst bucket is not touched.

I've implemented the compression now, but I may not deploy the change to our production system within the next two weeks.

Thank you

Rob

[Luke Bakken]

Hey Rob,

Thanks for that information. What does CPU use look like during one of these spikes? During a spike, does the iostat command or associated monitoring show high wait values?

There's a chance that some VM subsystem tweaking could help. These settings will cause the Linux kernel to flush data more frequently and in an async manner. The default settings cache quite a bit of data then flush it to disk at once which can block I/O:

https://docs.riak.com/riak/kv/latest/using/performance/index.html#optional-i-o-settings

Implementing compression should help greatly. Did you have any questions about my recommendation to use publisher confirms with an outstanding number of confirms to keep publishing throttled?

Thanks -

Luke

[Rob A.]

Hi Luke,

During the spikes our RabbitMQ uses about 3.2 CPU cores of max. 8 cores.

In the first queue the messages are still medium size (~3MB).

Then the message will be transmitted to another queue (workflow engine queue) with the large message size (~20 MB).

Afterwards the message will be moved to a 3rd queue and then moved back to the workflow engine queue.

So maybe we reached the CPU limitation of 1 core per queue. Is I/O wait part of the CPU core per queue limit?

Garbage collection may run on a different core, right?

Currently we throttled our batch process in production to stabilize the system.

Until next week we will build up a load test system to analyze the current issues (including the CPU usage and iostat) and compare the results with our changes.

Throttling with publisher confirms sounds like a good idea for our workflow engine.

But our consumers are only processing one request per instance, as they are slow and not multi-threading capable.

Therefore publisher throttling wouldn't work for them.

We have up to 120 consumer instances running during high load.

Besides throttling, wouldn't publisher confirms increase the memory usage, because messages are written in batches to disc?

Thank you

Rob

[Luke Bakken]

Hi Rob,

There's a lot of information and questions here so I'll answer in-line ...

During the spikes our RabbitMQ uses about 3.2 CPU cores of max. 8 cores.

In the first queue the messages are still medium size (~3MB).

Then the message will be transmitted to another queue (workflow engine queue) with the large message size (~20 MB).

Afterwards the message will be moved to a 3rd queue and then moved back to the workflow engine queue.

So maybe we reached the CPU limitation of 1 core per queue.

Yes, this is what is happening. A queue is a concurrency point within RabbitMQ.

 

Is I/O wait part of the CPU core per queue limit?

Not necessarily. I suspect you're CPU bound due to the message size and by having only 3 queues. Still, it would be interesting to see what the operating system reports via iowait during a spike, and if the tuning suggestion I made provides any benefit.

If it is possible to spread messages among more queues it should help this situation. I don't know what your message ordering requirements are, however. There are plugins to distribute messages between queues like the following:

https://github.com/rabbitmq/rabbitmq-sharding

https://github.com/rabbitmq/rabbitmq-consistent-hash-exchange

 

Garbage collection may run on a different core, right?

GC within the Erlang VM does not happen on a separate OS thread, I believe.

 

Throttling with publisher confirms sounds like a good idea for our workflow engine.

But our consumers are only processing one request per instance, as they are slow and not multi-threading capable.

Therefore publisher throttling wouldn't work for them.

Publisher confirms have nothing to do with consumers as they are for publishing messages only. Consumer acknowledgements are a different thing altogether and are what tells RabbitMQ that it is OK to consider a message successfully delivered to a consumer.

 

Besides throttling, wouldn't publisher confirms increase the memory usage, because messages are written in batches to disc?

Publisher confirms do not have anything to do with writing messages to disk. RabbitMQ may send confirms back to a publisher in a batch if the time taken to write the messages to all queues (and the disk) fall within a time window. Since you're dealing with large messages, your publishers will probably not see a confirmation for multiple messages, but you still should take that into account in your code.

Thanks -

Luke