memory consumption problem on quorum queues with big messages

[Stanislav K.]

We have a cluster of 3 nodes running quorum queues on AWS linux instances with 8GB RAM where we are experiencing problems with memory going over limit when pushing messages into queue and when growing quorum queues in case a node crashes and another node replaces it.

Our messages are quite big – about 5MB - with a rate of about 4/s.

Situation is, that producers push messages into queue without a consumer so a bunch of messages are collecting in a queue (e.g. 5000).

Occasionally, we are experiencing a problem, that a memory on some of the nodes goes over limit and does not go down, which blocks producers.

Another problem we have is, that if there are e.g. 5000 messages 5MB big in the quorum queue and one of the nodes crashes, we are starting a new node, adding it to cluster and then running grow command to add it to quorum. This always results in memory on the attaching node going over limit (e.g. 3-4 times higher than limit) and usually crashing the node (probably due to exhausting the node memory).

We tried tuning what we think are memory related parameters, without luck. Currently, following parameters are used in rabbitmq.conf

disk_free_limit.absolute = 25GB
vm_memory_high_watermark.absolute = 1500MB
mirroring_sync_batch_size = 10
raft.wal_max_batch_size = 10
raft.wal_max_size_bytes = 64000000

 I replicated the problem into simplified scenario with 3 nodes running on a single computer.

Rabbitmq 3.11.2, Erlang 25.1.1, 32G RAM, Windows 10, rabbitmq.conf as specified above

Scenario 1:

1. Set up 3 nodes cluster (e.g. rabbit7@localhost, rabbit8@localhost, rabbit9@localhost)
2. Create a quorum queue on that cluster (e.g. queue1)
3. Run perf test with 1 publisher 0 consumers 5MB messages size 5000 messages into that queue (e.g. java -jar perf-test-2.19.0.RC1.jar --producers 1 --consumers 0 --exchange amq.default --routing-key queue1 --skip-binding-queues --predeclared --pmessages 5000 --size 5000000 --uri amqp://localhost:5677)
4. Notice that occasionally RAM on a random node goes over limit, pausing perftest and it does not go down (force_gc on given node helps to resolve the problem)

Secnario 2:

1. Cluster setup as in previous scenario containing 5000 messages of size 5MB in a quorum queue (e.g. queue1)
2. Shut down any of the nodes in cluster (e.g. rabbit9@localhost). Remove that node from cluster
3. Start a new node (e.g. rabbit2@localhost) and add it to cluster
4. Run rabbitmq-queues grow command (e.g. rabbitmq-queues -n rabbit7@localhost grow rabbit2@localhost all)
5. Notice that eventually RAM on the added node (rabbit2@localhost) goes over limit and stays like this (force_gc resolves the problem). On our AWS (latest tested with Rabbitmq 3.11.0) grow command often consumes all node memory and crashes the node (node 8GB, vm_memory_high_watermark.absolute=15000MB, we have seen rabbitmq consume 6.7GB RAM)

During high memory situation, when I open Memory details of the node, I can see that almost all the memory is in Binaries and when opening Binary references, Quorum Queues/quorum is the one consuming almost everything (e.g. 2.2 GiB quorum).

Are there any other parameters besides those stated above that affect memory consumption that we could use to tune rabbitmq in order to resolve the problems that we are experiencing or is this a possible bug? I searched through mailing list as well as github issues but did not find description of similar problem especially connected to quorum grow command (scenario 2).

 Any help is appreciated.

 Stano

[kjnilsson]

Hi,

I think in this case we're hitting roughly the same issue, ingress rate is so high it cause a lot of binary data to be promoted the erlang process' old heap and once activity stops on the queue no further garbage collections will be automatically scheduled. A couple of comments:

Scenario 1: your perftest flags don't represent your expected ingress rate of 4/s. Try the test with the -R 4 and see if it behaves better.

Scenario 2: I'm looking at some options to reduce the pipeline buffers size for log replication which I believe is causing the new follower to also overflow. There is a parameter but it isn't configurable in RabbitMQ as it stands.

Also

raft.wal_max_batch_size = 10

This will keep WAL peak memory use down somewhat but it may also reduce the WAL throughput causing the quorum queue processes to backlog more. Can you run some tests with this substantially increased or just reset to default?

I'll get back to you when we have something further.

Cheers

Karl

[Stanislav K.]

Hi,

thanks for response. It is much appreciated.

Scenario 1: the --rate 4 was intentionally left out. With 4/s, AWS cluster works fine sometimes for days or even weeks, but occasionally RAM goes over limit and then manual intervention is required. I rerun perftest (on local cluster) with 4/s and with this limit I did not encounter RAM overflow. Without rate limit, I was able to reproduce it always therefore I left it out for easy demonstration.

I rerun both scenarios (scenario 1 without rate limit) with raft.wal_max_batch_size removed (deleted from rabbitmq.conf). The situation was the same. In fact, we started without any raft.*, mirroring_sync_batch_size or vm_memory_high_watermark.absolute used. We just added those config lines later in our attempts to solve memory problem in scenario 2.

Regards,

Stano

[Luke Bakken]

Hi Stanislav -

I'm sure you've already done this but if you can, compress your messages.

[Stanislav K.]

Hi,

yes, we are already applying compression before putting messages to queue.

Stano

[kjnilsson]

Just a quick note to say I am currently working on this issue to provide a few more configuration parameters that you can use to tune such large message workloads better.

I'll let you know when I have something that can be tested.

Cheers

Karl

[kjnilsson]

Ok so we have some experimental changes and new configuration parameters you can try. They are all in the ra application so you will have to use the advanced config option for this.

In particular I think the case where you grow a queue with 5k 5MB messages in it benefit from a much much lower `default_max_append_entries_rpc_batch_size` value (default is 128).

I have been testing with 16 and the memory use was a lot more stable than with 128. 

https://hub.docker.com/layers/pivotalrabbitmq/rabbitmq/rabbitmq-a746110-ra-d2fa989-otp-max/images/sha256-9177e98a163fbef19cc76e06644945d4d1f13a12a32645e66aed17cdb6e6d206?context=explore

For the case where you overrun the broker with data on publishing I think for now you are just going to have to throttle at the client level, I tested with a perftest -c flag of 8 and that gave me a stable 100 msg/sec with 5MB messages and decent memory profile.

At some point in the future we may implement a byte sized based flow control mechanism but right now it is done per message and is typically tuned for smaller message sizes.

A bigger point is whether I would recommend using queues for your 5MB messages, personally if messages were bigger than, say, 256KB I would probably consider using streams instead if at all possible.

Cheers

Karl

[kjnilsson]

For context here is the PR where the work is on going: https://github.com/rabbitmq/ra/pull/321

[Stanislav K.]

Thanks a lot for the updates.

Can you give me a hint how to run this version on windows without docker? Should I build from source?

Stano

[Stanislav K.]

I compiled raft(ra) from given branch (moar-configs) and replaced that in my rabbitmq 3.11.2 installation. I will do the tests with that and report on status.

[Stanislav K.]

So I run multiple tests including quorum grow under load (producers and consumers running with ~5k 5MB messages already inside queue). Playing with default_max_append_entries_rpc_batch_size did not bring too much so I just tried to guess all the parameters that might help. I ended up with this configuration.

vm_memory_high_watermark.absolute = 3GB

+

[
    {ra,[
        {wal_max_size_bytes, 64000000},
        {default_max_append_entries_rpc_batch_size, 4},
        {default_max_pipeline_count, 64},
        {segment_max_entries, 64},
        {wal_max_batch_size, 64}
    ]}
].
Given above quite did it. I no longer experienced memory spikes even to red. Please, notice I increased memory limit from 1,5GB to 3G. I did this after I noticed there are no extreme spikes of 3-4x the memory limit as experienced previously. I'm quite happy with how that worked during my local testing. Thanks a lot for your help and good work. Since I only guessed parameters that might affect memory consumption, please, let me know, if anything from advanced config sounds stupid in this context.

Stano