RabbitMQ Crash while importing 8MB file

[Kushagra Bindal]

Hi,

I am trying to upgrade my RabbitMQ fro 3.6.10 to 3.7.17 version. I am trying to upload an 8MB file generated from 3.6.10 version in a newly deployed RabbitMQ server of version 3.7.17. During performing this activity I am getting below error. Can someone please help in resolving this issue.

2019-09-30 14:34:31.718 [error] <0.6286.36> CRASH REPORT Process <0.6286.36> with 0 neighbours crashed with reason: no match of right hand value {more,<<"{\"rabbit_version\":\"3.6.10\",\"users\":[{\"name\":\"timekeeping07-b\",\"password_hash\":\"iZdDG98MfL2kByzVXFLLUvLZWunB9eFpvxLN7nLtNxlPBebe\",\"hashing_algorithm\":\"rabbit_password_hashing_sha256\",\"tags\":\"administrator\"},{\"name\":\"atul-srivastava\",\"password_hash\":\"ekF0BEy74IrJAth0++KrV+d/e2T6+2adfjWOukIaOMUWp4Sf\",\"hashing_algorithm\":\"rabbit_password_hashing_sha256\",\"tags\":\"administrator\"},{\"name\":\"johny-kandhari\",\"password_hash\":\"b0QvnyVUWKkW7lTCpBgDIQCkbvgP28Jw+QbanlTDlcuDpOdE\",\"hashing_algorithm\":\"...">>,...} in rabbit_mgmt_wm_definitions:get_all_parts/2 line 272

2019-09-30 14:34:31.719 [error] <0.6285.36> Ranch listener rabbit_web_dispatch_sup_15672, connection process <0.6285.36>, stream 1 had its request process <0.6286.36> exit with reason {badmatch,{more,<<"{\"rabbit_version\":\"3.6.10\",\"users\":[{\"name\":\"timekeeping07-b\",\"password_hash\":\"iZdDG98MfL2kByzVXFLLUvLZWunB9eFpvxLN7nLtNxlPBebe\",\"hashing_algorithm\":\"rabbit_password_hashing_sha256\",\"tags\":\"administrator\"},{\"name\":\"atul-srivastava\",\"password_hash\":\"ekF0BEy74IrJAth0++KrV+d/e2T6+2adfjWOukIaOMUWp4Sf\",\"hashing_algorithm\":\"rabbit_password_hashing_sha256\",\"tags\":\"administrator\"},{\"name\":\"johny-kandhari\",\"password_hash\":\"b0QvnyVUWKkW7lTCpBgDIQCkbvgP28Jw+QbanlTDlcuDpOdE\",\"hashing_algorithm\":\"rabbit_password_hashing_sha256\",\"tags\":\"administrator\"},{\"name\":\"yi-wang\",\"password_hash\":\"sXQTUgIprmTVEWaroC5j+zTZTATBu0DDelGznPJrqe0bTLay\",\"hashing_algorithm\":\"rabbit_password_hashing_sha256\",\"tags\":\"administrator\"},{\"name\":\"v05d00d00-a\",\"password_hash\":\"F0B+rTqU2nitboSM24blXcmZyC5H8cKTAlfh0qeqvn2R+mr6\",\"hashing_algorithm\":\"rabbit_password_hashing_sha256\",\"tags\":\"administrator\"},{\"name\":\"keng03-dev01-ins02-wfm63-fnt.int.dev.mykronos.com\",\"password_hash\":\"5e6TRDmU/6E3aHi8sar7R5apsfoMcgYPv5tDv+rn2PLv/S5W\",\"hashing_algorithm\":\"rabbit_password_hashing_sha256\",\"tags\":\"administrator\"},{\"name\":\"dcs04-b\",\"password_hash\":\"owQKlOuPe9dsRFu7jO3mufoMEImNIw03s9tjIUc9bPzeANaF\",\"hashing_algorithm\":\"rabbit_password_hashing_sha256\",\"tags\":\"administrator\"},{\"name\":\"maryia-nabzdorava\",\"password_hash\":\"NnSAdpGZqGEvzV2lpJ4id2FKDm9vZod6HrGMN1O8OSfhBd/K\",\"hashing_algorithm\":\"rabbit_password_hashing_sha256\",\"tags\":\"administrator\"},{\"name\":\"sabrina-liu\",\"password_hash\":\"QSTfqTicGZxsSyWCKP9uoQFmeKkf2WVteXKGxWRb1xETkbM4\",\"hashing_algorithm\":\"rabbit_password_hashing_sha256\",\"tags\":\"administrator\"},{\"name\":\"ryan-wannop\",\"password_hash\":\"7o2sE6HxYb5Tphsv9+QAZQ//zX0hlz8waK0FUuP8TulUMniH\",\"hashing_algorithm\":\"rabbit_password_hashing_sha256\",\"tags\":\"administrator\"},{\"name\":\"prashant-vasudev\",\"password_hash\":\"k48S78Vwhu01SBjJxiGv90RSUSddcnUXdVki8ccuukqofN2c\",\"hashing_algorithm\":\"rabbit_password_hashing_sha256\",\"tags\":...">>,...}} and stacktrace [{rabbit_mgmt_wm_definitions,get_all_parts,2,[{file,"src/rabbit_mgmt_wm_definitions.erl"},{line,272}]},{rabbit_mgmt_wm_definitions,accept_multipart,2,[{file,"src/rabbit_mgmt_wm_definitions.erl"},{line,127}]},{cowboy_rest,call,3,[{file,"src/cowboy_rest.erl"},{line,1576}]},{cowboy_rest,process_content_type,3,[{file,"src/cowboy_rest.erl"},{line,1100}]},{cowboy_rest,upgrade,4,[{file,"src/cowboy_rest.erl"},{line,288}]},{cowboy_stream_h,execute,3,[{file,"src/cowboy_stream_h.erl"},{line,296}]},{cowboy_stream_h,request_process,3,[{file,"src/cowboy_stream_h.erl"},{line,274}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]

[Luke Bakken]

Hi Kushagra,

It appears you've found a bug in this code - https://github.com/rabbitmq/rabbitmq-management/blob/v3.7.x/src/rabbit_mgmt_wm_definitions.erl#L272

If possible, would you mind using the "Reply privately to author" feature of google groups to send me your definitions file so I can reproduce it? It would greatly expedite a fix.

"Reply privately to author" is accessed via the drop-down menu on the right-hand side of this message.

I have filed an issue here if you would like to follow along - https://github.com/rabbitmq/rabbitmq-management/issues/739

Thanks,

Luke

[Luke Bakken]

Hello again,

Could you please provide the exact command or process you're using to upload the definitions? The reason I ask is that this bug is only hit if the Content-Type header is multipart/form-data. If you can set that header to application/json, you should be able to work around this problem.

Thanks,

Luke

[Kushagra Bindal]

Hi Luke,

As told by you, I send you a private reply. Please let me know if you receive the required details.

[Kushagra Bindal]

Hi Luke,

Thanks for your response.

I have few queries around the same.

1. Is there any size limitation on RabbitMQ side, that it can only support that much definitions only?

2. Is there any specific documentation available on RabbitMQ site for the same? There should be some specific documentation available for this about the limit of the system.

3. If this is a defect then is there any plan or defect raised for the same and what are the future plan for fixing this issue?

4. Isn't there any multi-threading mechanism available that can process that much amount of data by paralleling the execution?

5. Is there any standard tool available which can logically break this definition file into chunks.

It will be a great help to get the answer of above queries as that will help us to proceed further with our future approach.

PS: This is an incremental file and may be this file size will become >15MB as well in near future. So, we have to plan it accordingly.

Regards

Kushagra

[Luke Bakken]

 

1. Is there any size limitation on RabbitMQ side, that it can only support that much definitions only?

No, it depends on the performance characteristics of your environment. When you import an enormous number of definitions at once (as you are trying), RabbitMQ has to create many entries in its internal database, and synchronize those entries in your cluster, as well as start up things like vhosts, queues, etc.

 

2. Is there any specific documentation available on RabbitMQ site for the same? There should be some specific documentation available for this about the limit of the system.

No, because it is unusual to try to import thousands of definitions at once.

 

3. If this is a defect then is there any plan or defect raised for the same and what are the future plan for fixing this issue?

I have fixed the immediate problem you reported no match of right hand value {more,... (https://github.com/rabbitmq/rabbitmq-management/pull/740). To have seen this error, you must have tried to import the definitions using the management web interface, or some other tool that set the content-type to multipart/form-data. The rabbitmqadmin tool does not use this content-type value.

I also provided the following instructions:

• Break up your definitions into a series of smaller imports
• Import while using application/json as the content-type. Using the rabbitmqadmin import command will use this content-type value.

4. Isn't there any multi-threading mechanism available that can process that much amount of data by paralleling the execution?

 

If you're already overloading your system by importing a lot of data "multi-threading" isn't going to magically fix it.

5. Is there any standard tool available which can logically break this definition file into chunks.

Yes, you can use something like jq from the command line. Every programming language can parse and write JSON as well.

Luke

[Kushagra Bindal]

Hi Luke,

Thanks for providing your response on my queries.

Please find my comments inline.

If I will break the file into multiple file, then also processing time of 18000 vhost will remain as it is and which will still lead into a big maintenance window in case of Blue/Green upgrade as currently we are facing this as well. So, a possible solution of this problem will be a great help!! Looking forward for some possible way out from RabbitMQ community to resolve this issue.

On Wednesday, October 2, 2019 at 8:06:48 PM UTC+5:30, Luke Bakken wrote:

 

1. Is there any size limitation on RabbitMQ side, that it can only support that much definitions only?

No, it depends on the performance characteristics of your environment. When you import an enormous number of definitions at once (as you are trying), RabbitMQ has to create many entries in its internal database, and synchronize those entries in your cluster, as well as start up things like vhosts, queues, etc.

<Kushagra>: Do you mean our environment specific configuration? I can understand that based on input data RabbitMQ has to perform certain operations at its side. I am sure RabbitMQ has published some performance stats as well regarding the increasing impact on the performance with the increasing definitions in a single file. So, do we have any data that indicate any performance stats from RabbitMQ side that shows how many cores and processors are required to process such kind of bulky files? What are the recommendation from RabbitMQ side in such cases. As every organization will be having heir specific use cases.

 

2. Is there any specific documentation available on RabbitMQ site for the same? There should be some specific documentation available for this about the limit of the system.

No, because it is unusual to try to import thousands of definitions at once.

<Kushagra> : This is our standard use case and this could be a problem that may arise by some other person in future as well.

 

3. If this is a defect then is there any plan or defect raised for the same and what are the future plan for fixing this issue?

I have fixed the immediate problem you reported no match of right hand value {more,... (https://github.com/rabbitmq/rabbitmq-management/pull/740). To have seen this error, you must have tried to import the definitions using the management web interface, or some other tool that set the content-type to multipart/form-data. The rabbitmqadmin tool does not use this content-type value.

I also provided the following instructions:

• Break up your definitions into a series of smaller imports
• Import while using application/json as the content-type. Using the rabbitmqadmin import command will use this content-type value.

<Kushagra>: Thanks for providing the fix, I will check by running it by setting content-type as application/json. 

4. Isn't there any multi-threading mechanism available that can process that much amount of data by paralleling the execution?

 

If you're already overloading your system by importing a lot of data "multi-threading" isn't going to magically fix it.

<Kushagra>:  Correct, so we are here saying that while importing the definitions RabbitMQ itself is fully occupying the system resources and hence multi-threading is not a possible option in this case?

 

5. Is there any standard tool available which can logically break this definition file into chunks.

Yes, you can use something like jq from the command line. Every programming language can parse and write JSON as well.

 

<Kushagra>: Yes, I can understand that we can parse it very well, but as the structure is little complex and having dependencies, so I just wanted to ask if there is any possible way to divide the code into logical chunk by any tool plugin provided by RabbitMQ for the same. I can give a try from my end, if there is no such solution available as of now and if jq can simply help then I will check that as well.

Luke

[Kushagra Bindal]

Hi Luke

Please let me know if any configuration details or any other statistics will be required for this from my side which we can use to make any theory out of it.

Looking forward to hearing from you.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/56220546-5d54-4bcd-b530-7dfd8ac4f8de%40googlegroups.com.

[Michael Klishin]

Kushagra Bindal:

Hi Luke,

Thanks for providing your response on my queries.

Please find my comments inline.

If I will break the file into multiple file, then also processing time of 18000 vhost will remain as it is and which will still lead into a big maintenance window in case of Blue/Green upgrade as currently we are facing this as well. So, a possible solution of this problem will be a great help!! Looking forward for some possible way out from RabbitMQ community to resolve this issue.

On Wednesday, October 2, 2019 at 8:06:48 PM UTC+5:30, Luke Bakken wrote:

 

1. Is there any size limitation on RabbitMQ side, that it can only support that much definitions only?

No, it depends on the performance characteristics of your environment. When you import an enormous number of definitions at once (as you are trying), RabbitMQ has to create many entries in its internal database, and synchronize those entries in your cluster, as well as start up things like vhosts, queues, etc.

<Kushagra>: Do you mean our environment specific configuration? I can understand that based on input data RabbitMQ has to perform certain operations at its side. I am sure RabbitMQ has published some performance stats as well regarding the increasing impact on the performance with the increasing definitions in a single file. So, do we have any data that indicate any performance stats from RabbitMQ side that shows how many cores and processors are required to process such kind of bulky files? What are the recommendation from RabbitMQ side in such cases. As every organization will be having heir specific use cases.

Exactly because no two definition files are identical and most environments do not have 18K virtual hosts,  such data is not collected

and would be of little use. Monitoring your nodes (see the Monitoring guide [1]) would make a lot of difference, every time.

 

 

2. Is there any specific documentation available on RabbitMQ site for the same? There should be some specific documentation available for this about the limit of the system.

No, because it is unusual to try to import thousands of definitions at once.

<Kushagra> : This is our standard use case and this could be a problem that may arise by some other person in future as well.

Large definition files are not unheard of but besides three closely issues in large chunked file upload handler,

this is not really a topic you'd see frequently on this list.

 

 

3. If this is a defect then is there any plan or defect raised for the same and what are the future plan for fixing this issue?

I have fixed the immediate problem you reported no match of right hand value {more,... (https://github.com/rabbitmq/rabbitmq-management/pull/740). To have seen this error, you must have tried to import the definitions using the management web interface, or some other tool that set the content-type to multipart/form-data. The rabbitmqadmin tool does not use this content-type value.

I also provided the following instructions:

• Break up your definitions into a series of smaller imports
• Import while using application/json as the content-type. Using the rabbitmqadmin import command will use this content-type value.

<Kushagra>: Thanks for providing the fix, I will check by running it by setting content-type as application/json. 

3.7.19 already includes it. IIRC it's made it into 3.8.0 as well but it also has a ton of other things so if you want to just try out the import endpoint,

stick to 3.7.19 for this test.

 

4. Isn't there any multi-threading mechanism available that can process that much amount of data by paralleling the execution?

 

If you're already overloading your system by importing a lot of data "multi-threading" isn't going to magically fix it.

<Kushagra>:  Correct, so we are here saying that while importing the definitions RabbitMQ itself is fully occupying the system resources and hence multi-threading is not a possible option in this case?

 

"Multi-threading" is not a magic pixie dust you sprinkle on a problem to make it go away. It's a design decision where you have to ask:

  * What are the bottlenecks in a specific workload

 * Can they be parallelised

 * Is the parallelisation going to be worth the synchronisation cost

 * Is the parallelisation going to be worth the code maintenance cost

In case of this endpoint, the issue is that starting 18K virtual hosts means starting at least 54K Erlang processes.

It takes time and primarily CPU resources. RabbitMQ's runtime will already use all the cores it detects by default,

and usually reasonably efficiently [2]. File size is irrelevant here.

Monitoring the node undergoing the import will provide a lot of relevant information. If the node maxes out all cores for a period of time

during the import, consider at least temporarily provisioning more CPU cores for it. On a system that's already busy serving traffic,

serious overprovisioning and importing smaller definition chunks might be necessary.

 

5. Is there any standard tool available which can logically break this definition file into chunks.

Yes, you can use something like jq from the command line. Every programming language can parse and write JSON as well.

 

<Kushagra>: Yes, I can understand that we can parse it very well, but as the structure is little complex and having dependencies, so I just wanted to ask if there is any possible way to divide the code into logical chunk by any tool plugin provided by RabbitMQ for the same. I can give a try from my end, if there is no such solution available as of now and if jq can simply help then I will check that as well.

Luke

There is no tool that would split a definition file into smaller chunks. The structure of a definition file is fairly straightforward:

 * Users

 * Virtual hosts

 * Permissions that connect the two  above

 * Runtime parameters

 * Exchanges

 * Queues

 * Bindings

I'm not even sure what else is there.

I'd focus on monitoring the node to see how close is it to maxing out on CPU and trying a few scheduler

flags mentioned in [2] to see if it reduces the amount of context switching performed. At least you'd be able to find

a point at which the import does not bring all other activity such as client operations to a crawl.

Or find the smallest  file that is imported without spiking the load for long enough to be problematic.

1. https://www.rabbitmq.com/monitoring.html

2. https://www.rabbitmq.com/runtime.html

[Kushagra Bindal]

Thanks for providing your response Michael.

I can very well understand that this is unique kind of use case and we have to support this in our environment as well. 

So, can you guys please suggest me the way to move ahead with this problem. While performing Blue/Green we have a defined maintenance window and we have to cope up with this time duration. So, what are possible ways which we can explore in this situation?

Looking forward for some possible way out from RabbitMQ community to resolve this issue.

Regards

Kushagra

[Luke Bakken]

Hi Kushagra,

You can use servers that have much higher performance than you currently use. That might work. Since importing definitions creates a lot of disk activity having fast disks is important.

You can use multiple RabbitMQ clusters and divide your vhosts among them. I'm assuming at this point you're using one RabbitMQ cluster.

In any case, you must monitor your system as Michael suggested so that you know if your changes are having a positive or negative effect.

Thanks,

Luke

[Kushagra Bindal]

Hi Luke/Michael,

Thanks for your response. I will give it a try and update you in case of any further concerns or queries.

Regarding: Crash for content-type to multipart/form-data,please let me know, if I need to upgarde it to latest version of RabbitMQ or I can patch your fix directly into my RabbitMQ cluster and can restart the system. If possible, then please let me know the step to patch the same.

Regards

Kushagra

[Michael Klishin]

While you can replace the management plugin in your installation with the one that ships with 3.7.19, we advise against it

as there can be subtle dependencies on core broker's internal API. Please upgrade.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/1440a251-6456-4ea4-87dc-48fb20d2dcf2%40googlegroups.com.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Luke Bakken]

Regarding: Crash for content-type to multipart/form-data,please let me know, if I need to upgarde it to latest version of RabbitMQ or I can patch your fix directly into my RabbitMQ cluster and can restart the system. If possible, then please let me know the step to patch the same.

Import your definitions using rabbitmqadmin and you won't see that error. 

[Kushagra Bindal]

Hi Luke/Michael,

Updating to 3.7.19 is not feasible for me right now. Is there anyother option available by which we can manually add these fixed files as a patch.

In addition to that I have tried rabbitmqadmin option that Luke suggest with relatively a smaller file of 3MB.

Below is the command that I used to execute the file of 3MB.

rabbitmqadmin -P 15672 -u mqowner -p mqowner --request-timeout 3600000 import DMQ25-INS.json

While executing the same it ran properly for 2-3 min and after that I got another error. Below is the error that I got while executing the above file.

*** Error getting HTTP response:

Traceback (most recent call last):

  File "/usr/sbin/rabbitmqadmin", line 587, in http

    resp = conn.getresponse()

  File "/usr/lib64/python2.7/httplib.py", line 1051, in getresponse

    response.begin()

  File "/usr/lib64/python2.7/httplib.py", line 415, in begin

    version, status, reason = self._read_status()

  File "/usr/lib64/python2.7/httplib.py", line 379, in _read_status

    raise BadStatusLine(line)

BadStatusLine: ''

Can you please check and let me know why this error is coming and what is workaround of this?

Regards

Kushagra

[Manish Kumar]

Just to add  -

We ran thisrabbitMQAdmin command multiple times with 3 MB import definition file and every time we are getting the same error, approx it ran for around 5 mins (each time ) and then failed with the error mentioned below. 

Manish

[Michael Klishin]

There was no response (no HTTP status line) when the client attempted to read the status.

My guess is that the client has more than one timeout but we only propagate just one.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/76e85960-664a-4a90-ad26-7fdcf8ee6509%40googlegroups.com.

[Michael Klishin]

You can try replacing rabbitmq-management plugin in your distribution with the one from 3.7.19 but

we cannot guarantee compatibility. Upgrading is a much better option.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/124770c5-de26-411c-b79b-b03ec5985d52%40googlegroups.com.

[Luke Bakken]

Hi Manish,

Most likely the import timed out somewhere. Please capture traffic on port 15672 and share the capture around the time the exception is raised.

There may also be something logged by RabbitMQ at the same time. Did you check the RabbitMQ log file? Please share the log file's contents from the same time period.

Thanks,

Luke

[Kushagra Bindal]

Thanks Luke. I'll share the logs details with you soon.

In the meantime can you please share the steps to upgrade rabbitmq management so that I can try by upgrading the same as well. 

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/42c0edeb-cc36-4555-96e4-49585f8c39d4%40googlegroups.com.

[Michael Klishin]

There  are only two options:

  * Upgrade RabbitMQ as a whole

  * or locate the rabbitmq-management-3.7.19.ez in the plugins directory of 3.7.19, copy it to the 3.7.17 plugins directory and restart the node

We do not guarantee that this method will work specifically in this case or in general. Plugins sometimes have to change in lock step with RabbitMQ core.

`rabbitmq-plugins directories` will display relevant directory information on a running node (useful to determine where to copy the file)

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CALGrN%2B87LLc541tnmkLfBptr1v8o-i-_%2BXe0nSPEReq5aG-yyw%40mail.gmail.com.

[Kushagra Bindal]

Hi Michael

Thanks for your response.

So you are saying that rabbitmq-management plugin does not requirss additional upgrade it ships with rabbitmq original image itself. As I have updated my rabbitmq from 3.6.10 to 3.7.17 so it is by default upgraded to 3.7.17??

I will give it a try by replacing 3.7.19.ez in rabbitmq-plugin directory. 

Can you please provide me the web link from where I can get 3.7.19.ez file.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CAGcLz6WX-pS9xYO1fZudk2Md-aB9_1SoZ04Yf4b%2BN6zgs0uNWQ%40mail.gmail.com.

[Michael Klishin]

Yes, RabbitMQ 3.7.17 distribution includes a management plugin suitable for it and of identical version.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CALGrN%2B_eAVmB1c5matnvyCuOShtb8BXhKH0q%2B%3D_UPtBJDzT6eg%40mail.gmail.com.

[Kushagra Bindal]

Thanks Michael

Can you please share 3.7.19.ez file location to me so that I can replace the same in my deployment. 

Is there anything specific that I need to take care of? 

Do I need to convert it to any specific format?

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CAGcLz6XYgUoDc1jx-vdYyKDh2Ko2%2BdbtCyxY14TMq_9NdYL-cw%40mail.gmail.com.

[Kushagra Bindal]

In addition to my previous email, just wanted to know that the main reason for your recommendation of using 3.7.19 version's rabbitmq-management plugin is because rabbitmq provides a fix for this http-response failure issue in 3.7.19 version??

[Michael Klishin]

The reason why I'm saying that a new plugin version (future 3.7.19) is necessary is because of [1]

and the fact that there is no workaround that 3.7.17 can use.

rabbitmqadmin is a zero dependency Python script that can be copied from GitHub and modified however

you see fit.

Here's a version that includes [1]. The link should be valid for seven days.

1. https://github.com/rabbitmq/rabbitmq-management/pull/740

2. https://send.firefox.com/download/c9c08eb7ee248b8c/#ruaaIxF-p8-A4syoSHWoIg

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CALGrN%2B-eN1%3D6YA2L44Yx%3DXOqjQWOQyN%2BFfU-bEx%3DxooHpw2sDA%40mail.gmail.com.

[Michael Klishin]

My apologies, I meant "a future 3.7.20" but since 3.7.19 has already shipped (I'm too focused on 3.8.1 development a.t.m.) and includes the PR above,

simply download the generic UNIX tarball, extract it and take a look under the ./plugins directory.

[Kushagra Bindal]

Hi Luke/Michael

We have already upgrade our environments to 3.7.17 and to qualify the upgrade process we had to perform multiple iteration of performance tests & functional qualification. Not only for Rabbitmq but for any tool/software as well it includes a huge efforts to upgrade the third party tool.

Whatif we will upgrade our version from 3.7.17 to 3.7.20. Does that resolve all the issues that we are currently experiencing with current version of Rabbitmq.

Or do you have any other suggestions on this. 

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CAGcLz6U9%3DzT2omGf6w3U%2BbqAVGjRYJdP64O%3DDEOe7dhNkm-Shg%40mail.gmail.com.

[Luke Bakken]

Whatif we will upgrade our version from 3.7.17 to 3.7.20. Does that resolve all the issues that we are currently experiencing with current version of Rabbitmq.

Or do you have any other suggestions on this

 It feels like we're repeating ourselves over-and-over at this point.

You are experiencing two different issues:

* The first issue, that you reported here is resolved by upgrading to RabbitMQ 3.7.20 when it ships, or by using the management plugin from 3.7.19 as Michael indicated. This error (no match of right hand value {more,<<"{\"rabbit_ve...) only manifests itself when you upload a definitions file using multipart/form-data as the Content-Type value. Note that this could not have happened using the rabbitmqadmin tool as that tool only uses application/json as the Content-Type value.

* The second issue, reported first in a private email to me, is due to the enormous size of your definitions. We have given a couple suggestions such as breaking up the definitions or breaking up your cluster so that you don't try to import many thousands of vhosts and other entities at once. You may be able to increase the computing power of your environment to work around this but the basic issue is that you're overloading your cluster with this import and a timeout is happening at some point during the process. We do not have a concrete "fix" for this other than the suggestions we have given.

Thanks,

Luke