Virtual host creation is slow in RabbitMQ version 3.8.24 as compared to 3.6.10

[aatish bansal]

HI,

I am importing a json file in RabbitMQ version 3.8.24

|  Component  |  Size         |

|   vhost            |  1750        |

|   queues         |  3612        |

|   exchanges   |  18485     |

Virtual host creation is slow as compared to version 3.6.10. The whole process takes around 18 mins on version 3.8.24 and same file took 2 min on rabbit 3.6.10 Is any thing handled on new version while creating vhost on latest version.

[Henrik Y]

Do you have any error log showing the case?

[aatish bansal]

There is no error regarding this. But we observe slowness in new version while importing a json.

[Hoffman S]

Does RMQ 3.8 use the latest version of Erlang?

> --
> You received this message because you are subscribed to the Google
> Groups "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to rabbitmq-user...@googlegroups.com.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/rabbitmq-users/5ccebcd0-3bde-419c-a2ac-d31bf4372572n%40googlegroups.com
> [1].
>
>
> Links:
> ------
> [1]
> https://groups.google.com/d/msgid/rabbitmq-users/5ccebcd0-3bde-419c-a2ac-d31bf4372572n%40googlegroups.com?utm_medium=email&utm_source=footer

[aatish bansal]

We are using erlang-23.3.4.8-1.el7.x86_64.rpm (23.3.4.8 version) erlang and rabbitmq-server-3.8.24-1.el7.noarch.rpm (3.8.24 version).

[Hoffman S]

I would suggest you try erlang 24.

regards.

> https://groups.google.com/d/msgid/rabbitmq-users/629a971f-63f7-4d92-9b9d-4f5cec60c226n%40googlegroups.com

> [1].
>
>
> Links:
> ------
> [1]

> https://groups.google.com/d/msgid/rabbitmq-users/629a971f-63f7-4d92-9b9d-4f5cec60c226n%40googlegroups.com?utm_medium=email&utm_source=footer

[aatish bansal]

Thanks @Hoffman for your reply.

We are centOS 7, so can you please share or suggest rpm file for erlang 24 and RabbitMQ 3.8.24 version

[aatish bansal]

Erlang 24 depends on OpenSSL 1.1, which is not available on CentOS 7. Therefore Erlang 24 packages are only produced for CentOS 8.

As suggested above 3.8.24 is compatible with 23.2 (erlang minimum version required), so we used 23.3.4.8. As erlang 24 version is not available for centOS 7.

[Hoffman S]

try packagecloud:

https://packagecloud.io/rabbitmq/rabbitmq-server
https://packagecloud.io/rabbitmq/erlang

Thanks.

> https://groups.google.com/d/msgid/rabbitmq-users/a9789a52-ed4a-4fbc-9bed-0f542aa3f160n%40googlegroups.com

> [1].
>
>
> Links:
> ------
> [1]

> https://groups.google.com/d/msgid/rabbitmq-users/a9789a52-ed4a-4fbc-9bed-0f542aa3f160n%40googlegroups.com?utm_medium=email&utm_source=footer

[aatish bansal]

Sorry for re-iterating. But erlang 24 version is not available for centos7.

https://github.com/rabbitmq/erlang-rpm

Regards,

[Loïc Hoguin]

Hello,

 

I do not think Erlang 24 is going to make 18 minutes back into 2 minutes.

 

There has been a few changes since 3.6.10. The simplest way to identify which version slowed everything down would be to pick a version in-between and import the definitions again to see if that version is affected. I would start with 3.7.0.

 

I would also double check that it is indeed vhosts: have you tried removing everything except vhosts from that file? Are you positive that they’re the issue? One thing that changed in 3.7.0 is that the message store is per vhost, meaning that there’s a need for more FDs, make sure you allow plenty of those.

 

It would be helpful if you could do the version testing and then open a ticket with a summary of the import times over a few versions. It would help solve this problem faster. We are aware that there are issues but solving them has not been prioritized.

 

If that’s not possible I would be interested in having the file nevertheless. Then we can use it when we get to work on these issues. You can provide it in a ticket or by email.

 

Cheers,

 

-- 

Loïc Hoguin

 

From: <rabbitm...@googlegroups.com> on behalf of aatish bansal <aatishba...@gmail.com>
Reply to: "rabbitm...@googlegroups.com" <rabbitm...@googlegroups.com>
Date: Tuesday 23 November 2021 at 09:57
To: rabbitmq-users <rabbitm...@googlegroups.com>
Subject: [Suspected Spam] Re: [rabbitmq-users] Virtual host creation is slow in RabbitMQ version 3.8.24 as compared to 3.6.10

 

Sorry for re-iterating. But erlang 24 version is not available for centos7.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/14deb512-69a7-44a7-8e2b-7465ac5fe73cn%40googlegroups.com.

[aatish bansal]

We  tried creating of vhost only.

Please find the stats for the same.

--------------------------------------------------------------------------
|  vHost   |  Current (3.6.10)   |  Latest (3.8.24)  |

--------------------------------------------------------------------------

|  100       |     3 sec                  |   15 sec                |

--------------------------------------------------------------------------

|  1600     |    1 min 11 sec     |   23 mins 7 sec   |

--------------------------------------------------------------------------

[Michal Kuratczyk]

Hi,

I can confirm that I see similar results and that a node restart is affected as well.

At first glance, the problem is related to Mnesia transactions/locking. We are investing in replacing Mnesia

altogether (with https://github.com/rabbitmq/khepri/), exactly because of issues like this. However, this issue

has almost certainly been there, unnoticed, since 3.7.0 (so almost exactly 4 years). You hit this issue because

you have a very unusual setup:

- it's the first time I see so many vhosts in a cluster

- on average, you have only 2 queues per vhost

- you have 5 times more exchanges than queues (usually the ratio is reversed, since a single exchange can route to many queues)

Can you provide some explanation for why you need a setup like this? Perhaps the easiest solution is for you to use

RabbitMQ in a more common way. Also, this issue is not linear - in my system 1000 vhosts takes 3 minutes but 2000

vhosts takes 13 minutes, so if you can reduce the number of vhosts by half for example, you should see a few times faster

import/node start.

We can have a look if there is something simple we can do to resolve this but we may decide to leave it as-is until we replace Mnesia.

Best,

On Tue, Nov 23, 2021 at 10:42 AM aatish bansal <aatishba...@gmail.com> wrote:

We  tried creating of vhost only.

Please find the stats for the same.

--------------------------------------------------------------------------
|  vHost   |  Current (3.6.10)   |  Latest (3.8.24)  |

--------------------------------------------------------------------------

|  100       |     3 sec                  |   15 sec                |

--------------------------------------------------------------------------

|  1600     |    1 min 11 sec     |   23 mins 7 sec   |

--------------------------------------------------------------------------

--

Michał

RabbitMQ team

[aatish bansal]

Hi Michal,

This is just our one environment details in which most of the queues got expired.

We are having ~30 queues per vhost and ~10-13 exchanges per vhost.

Regards,

Aatish Bansal

[Michal Kuratczyk]

Ok, but what do you need 1750 vhosts for? :) If we are to put effort into fixing this, we would like to understand why this is even needed.

Is this a cluster for test environments and each environment is a separate vhost or do you use that many vhosts (and as I understand ~60k queues) in production?

If the latter, what is the rationale for splitting queues into so many vhosts?

Thanks,

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/f08cfe27-8b65-4758-8750-d553264c62c4n%40googlegroups.com.

--

Michał

RabbitMQ team

[aatish bansal]

Sorry Michael for the late reply.

 

Yes this cluster is used by developers and testers combined and in general there are 1500+ vhosts. We need to upgrade this environment also with latest RabbitMQ.

We have these many vhosts and queues in this environment because we are highly dependent on RabbitMQ for frontend and backend calls.

So please provide the solution to this issue. 

I have observed as the number of vhost starts increasing, the time to creation also proportionally increases.

Also with these numbers, restarting of nodes is also taking time (nearly 1 hour for each node) on 3.8.24 while on 3.6.10, it takes ~20 sec.

Thanks,

Aatish Bansal 

[Michal Kuratczyk]

Hi,

I had a look at this issue and I'm fairly sure the problem is related to the number of exchanges declared (at least to some extent in parallel) - each vhost contains 7 default exchanges and they all get declared when a vhost is created/started. With almost 2000 vhosts, you have almost 14000 exchanges. I'm fairly sure it could be improved, although it can be hard to make it as fast as it used to be. But to be honest, it's not exactly a top priority - it's the first time we hear about RabbitMQ with more than a few hundred vhosts and hundreds work reasonably well (and that's already beyond what vhosts were meant for). My question still stands - if you rely on RabbiMQ so heavily and you need thousands of test environments, then why don't you just deploy a second cluster? We built the Kubernetes Operator to deploy clusters easily. It's also very easy to run RabbitMQ locally. There will always be limits to what a single cluster can handle - it can be the number of vhosts, number of exchanges, number of queues, number of messages, disk throughput, or something else. You can't assume that you'll always be able to just add more stuff to a single cluster. And if you really need such an usual configuration, it feels like the perfect opportunity to make an open source contribution  - we'd certainly welcome a pull request which improves the performance of this unusual use case.

Best,

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/f2d0f361-f52e-4d8e-8de0-45538ae1614en%40googlegroups.com.

--

Michał

RabbitMQ team

RabbitMQ team

[aatish bansal]

Hi Michael,

Thanks for your quick response.

Based on current situation, please suggest a workaround for this issue.

In general our expectation with version upgrade is to enhance performance of RabbitMQ and atleast, it should handle  our present load with similar infrastructure (if not less).

Thanks,

Aatish Bansal

[Michal Kuratczyk]

I've already suggested three workarounds/solutions:

- use fewer vhosts

- use more clusters

- contribute an improvement (I can give you some pointers if you want)

I understand that this is a regression compared to 3.6 but you have a very unusual requirement that we don't test or optimize for and everything is a trade-off - changes we've made to optimize much more common use-cases unfortunately caused this usunal one to get worse.

Best,

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/0cedce9d-931a-42d2-a4b7-3978486fcdacn%40googlegroups.com.

--

Michał

RabbitMQ team