Starting rabbitmq times out when many durable queus

[jacobh...@gmail.com]

We are using RabbitMQ with many durable queues over 200K queues, and when we want to restart the service it times out and fails to start, the only way to solve is to delete the mnesia folder.

Is there some configuration that can help us have so many durable queues and successfully restart the service ?

[Wes Peng]

1. Keep queue small, consume  out messages as soon as possible

2. Use fast disk such as nvme. No raid needed if you have multiple replications.

3. Upgrade RMQ to the latest version

Thanks 

On Mon, Mar 7, 2022 at 5:21 AM jacobh...@gmail.com <jacobh...@gmail.com> wrote:

We are using RabbitMQ with many durable queues over 200K queues, and when we want to restart the service it times out and fails to start, the only way to solve is to delete the mnesia folder.

Is there some configuration that can help us have so many durable queues and successfully restart the service ?

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/f86d32bb-9c3a-475a-bdec-f2e57c61afaan%40googlegroups.com.

[jacobh...@gmail.com]

Thank you for the quick response

1. Our queues are very small, just  a few KB or less

2. We are using AWS r5.4xlarge with EBS volume, we will try to change to ssd or nvme

3. We are using  RMQ 3.9.8

[Michal Kuratczyk]

Hi,

Would you mind sharing:

1. your definitions file (rabbitmqctl export_definitions) or at least one queue definition if they are all the same?

2. node startup logs, ideally at debug level

Also, if you have a definitions JSON file specified in your config file (load_definitions) then:

1. hopefully you can remove it to avoid reimporting queues you already have defined

2. In 3.10, there will be an option to skip importing previously imported definitions file (based on a checksum, so "skip import if the JSON file didn't change" options, see: https://github.com/rabbitmq/rabbitmq-server/blob/master/release-notes/3.10.0.md)

Best,

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/dc728774-28d7-46c4-89d4-9b3b8b09b18bn%40googlegroups.com.

--

Michał

RabbitMQ team

[jacobh...@gmail.com]

1. queue definition ( all are the same ) :
"queues": [
                {
                        "arguments": {},
                        "auto_delete": false,
                        "durable": true,
                        "name": "someid",
                        "type": "classic",
                        "vhost": "/"
                },

2. We dont have a definitions JSON file.

3. I cant currently share the log files, is there something specific I should try to look for ?

[Michal Kuratczyk]

Thanks, I'll define 200k of these and see what I get. Just to confirm:

1. Are there any policies in effect for these queues?

2. These queues are pretty much empty, correct?

The logs would show which steps of the boot process consume the time. If you can at least have a look at the logs and tell us which lines are "far apart" (time-wise), that'd be helpful.

At debug level, you should also see lines such as

- Time to start RabbitMQ ...

- Recovering N queues of type rabbit_classic_queue took Xms

- rabbit_binding:recover/2 for vhost / completed in...

Lastly, how long does it take before RabbitMQ gives up booting?

Best,

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/94705887-b97a-4054-88e5-4cf04e4c1f61n%40googlegroups.com.

--

Michał

RabbitMQ team

[jacobh...@gmail.com]

Thanks

1. We don't have policies on theses queues

2. Most are empty, some have just a few KB ( is there a simple way to check this at runtime for so many queues ? )

I don't have the exact information because this is a production environment and reproducing means downtime...

[Michal Kuratczyk]

Hi,

On my test machine, a node with 100k classic queues starts in about 10 minutes. With the upcoming v2 of the classic queues, it goes down to 6 minutes.

How long does it take for you? Any chance for some info from your logs? Bindings, especially topic bindings, can also take a long time to import/load on boot

so it could be that the culprit is not with the queues, but rather the bindings.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/adbd6cd4-d3df-45d6-84ea-8ba6c0375487n%40googlegroups.com.

--

Michał

RabbitMQ team

RabbitMQ team