connection to rabbitmq via amqp takes too long

[stuf...@gmail.com]

i`ve got rabbitmq running on centos 7 in vmware.

Connecting to it locally from scripts running on centos.

Connection takes a very long time (8-9 seconds).

i`ve been trying to connect from php-amqp, or from node.js - result is the same.

And it is the same time that it takes when issuing any command to rabbimqctl.

for example, list_queues.

i`m running a completely clean installation, version 3.3.5 (set up via yum)

with absolutely no data and no load.

all configuration as-is.

There is absolutely nothing strange in rabbitmq logs.

[Michael Klishin]

I have a few hypotheses:

* OS (guest or even host) swapping
* Tooling such as SELinux (typical on RHEL) or anti-viruses (less typical) have an effect.

DNS resolution that takes a long time or (e.g. a server times out) may be an issue, although if you only have 1 node on localhost it, it should not go through
remote machines to perform resolution.

Can you capture localhost traffic with Wireshark and post a dump here?

MK

[stuf...@gmail.com]

1. not sure about swapping, and don`t know, how to check it.

2. SELinux would either restrict access or grant it.

      Not something in the middle like wait for 8 seconds then go))

      Nevertheless, as soon as it is a development machine, SELinux is disabled, no antiviruses are installed.

3. I`ve had very similar problem with big ssh delay when connecting to this centos server from a remote machine.

      Don`t know, if it`s somehow connected to current issue, but it was solved by setting up "UseDNS no" in sshd.conf.

      So, probably, i really got some problems with SOME dns resolutions (maybe, with resolving host name of a local machine)

      but, don`t know how to fix it. (i`ve set it up in hosts file, and found no other advise on the internet to fix it further)

      And resolving all remote names goes fast enough.

4. Capture file from wireshark is attached, and here is the timing output of my subscription script,

      that was executed while sniffing.

      (i`ve insert time tracking for every significant event):

[user@centos7 server]$ php page_parser.php
RCMLibs\AMQPConsumer constructed at 1,423,727,121.852747 seconds
359992 memory already used
connected 1 items processed for 7.992251 seconds
using 669624 of memory
got new channel 2 items processed for 7.995181 seconds
using 910624 of memory
got exchange 3 items processed for 7.996156 seconds
using 910912 of memory
got queue 4 items processed for 7.997243 seconds
using 911208 of memory
waiting for messages
channel subscribed 5 items processed for 7.998716 seconds
using 911992 of memory
^C

четверг, 12 февраля 2015 г., 9:10:02 UTC+3 пользователь Michael Klishin написал:

[Michael Klishin]

 On 12 February 2015 at 11:23:32, stuf...@gmail.com (stuf...@gmail.com) wrote:
> 1. not sure about swapping, and don`t know, how to check it.

iostat in both guest and host (if you have access to it)

> 3. I`ve had very similar problem with big ssh delay when connecting
> to this centos server from a remote machine.
> Don`t know, if it`s somehow connected to current issue, but it
> was solved by setting up "UseDNS no" in sshd.conf.

This is certainly a good indication that investigate the hostname resolution hypothesis is a good idea. 
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

On 12 February 2015 at 11:23:32, stuf...@gmail.com (stuf...@gmail.com) wrote:

> 4. Capture file from wireshark is attached, and here is the timing
> output of my subscription script,
> that was executed while sniffing.

So indeed it seems to take almost 7 seconds for server to send connection.start after the client has sent a protocol header
(timestamps: 13.14, then 21.07). This is odd and I'm guessing indicates a timeout happening somewhere.

The client connects over IPv6 — is this intentional?

I'd try with 3.4.4 and IPv4 to compare (you can configure RabbitMQ to listen only on IPv4 endpoints, see
tcp_listeners on [1][2]).

1. http://www.rabbitmq.com/configure.html
2. http://hg.rabbitmq.com/rabbitmq-server/file/4f986b94ac70/docs/rabbitmq.config.example#l22

[Michael Klishin]

On 12 February 2015 at 11:40:03, Michael Klishin (mkli...@pivotal.io) wrote:
> So indeed it seems to take almost 7 seconds for server to send
> connection.start after the client has sent a protocol header
> (timestamps: 13.14, then 21.07)

For those following along at home, the display filter I've used is "tcp.port eq 5672", nothing fancy. 

[Michael Klishin]

 On 12 February 2015 at 11:41:58, Michael Klishin (mkli...@pivotal.io) wrote:
> For those following along at home, the display filter I've used
> is "tcp.port eq 5672", nothing fancy.

More ideas: run your PHP script under strace.

Also see [1], you have discovered their item #1, see also #2.

HTH. 

1. http://injustfiveminutes.com/2013/03/13/fixing-ssh-login-long-delay/

[stuf...@gmail.com]

Is that creepy matryoshka-style config file works in the first place?

I`m not familiar with erlang, but my ide highlights error after the first '[' symbol, saying that '{' is unexpected...

and rabbitmqctl status shows warning:

WARNING: ignoring /etc/rabbitmq/rabbitmq.conf -- location has moved to /etc/rabbitmq/rabbitmq-env.conf

i`ve just uncommented object(?) with tcp_listeners in it. Precisely, its ipv4 part.

My version attached.

четверг, 12 февраля 2015 г., 1:17:10 UTC+3 пользователь stuf...@gmail.com написал:

[stuf...@gmail.com]

Also, i`ve noticed, that, unlike all other programs, rabbitmq-server starts / stops / restarts for too long.

For example, systemctl stop rabbitmq-server takes about 18 seconds.

And  systemctl start rabbitmq-server takes about 35 seconds 

[Michael Klishin]

 On 13 February 2015 at 00:31:06, stuf...@gmail.com (stuf...@gmail.com) wrote:
> Is that creepy matryoshka-style config file works in the first
> place?

The example one? Yes, it is syntactically correct and loads just fine .

> I`m not familiar with erlang, but my ide highlights error after
> the first '[' symbol, saying that '{' is unexpected...
> and rabbitmqctl status shows warning:
> > WARNING: ignoring /etc/rabbitmq/rabbitmq.conf -- location
> has moved to /etc/rabbitmq/rabbitmq-env.conf

It should be /etc/rabbitmq/rabbitmq.config. /etc/rabbitmq/rabbitmq-env.conf is for environment
variables.

You also don't have to copy and edit the entire example. Voilà:
https://gist.github.com/michaelklishin/f6a07d13f33232c985cb.

[Michael Klishin]

I highly recommend investigating what's going on with your DNS resolver and other parts
of the environment.

Things don't take 18 to 35 seconds for no reason. The same goes for your client connections
on localhost and inbound SSH connections (per your own words) on the very same machine.

RabbitMQ certainly performs DNS lookups on start. 

[stuf...@gmail.com]

Thanks for the short version.

It should be /etc/rabbitmq/rabbitmq.config. /etc/rabbitmq/rabbitmq-env.conf is for environment
variables.

But, thats exactly where i am putting it.

/etc/rabbitmq/rabbitmq.config

Thats why i`ve posted this strange warning.

i`ve got no file

/etc/rabbitmq/rabbitmq-env.conf

[stuf...@gmail.com]

oops.

Nope, that`s my mistake:

i`ve named it rabbitmq.conf instead of rabbitmq.config.

Sorry)

[Michael Klishin]

 On 13 February 2015 at 09:51:21, stuf...@gmail.com (stuf...@gmail.com) wrote:
> But, thats exactly where i am putting it.
> /etc/rabbitmq/rabbitmq.config
> Thats why i`ve posted this strange warning.
> i`ve got no file
> /etc/rabbitmq/rabbitmq-env.conf

The warning say

> WARNING: ignoring /etc/rabbitmq/rabbitmq.conf -- location has moved to /etc/rabbitmq/rabbitmq-env.conf

It should be .config, not .conf. RabbitMQ assumed you've meant /etc/rabbitmq/rabbitmq-env.conf
(some code archeology suggests /etc/rabbitmq/rabbitmq.conf was used instead of rabbitmq-env.conf in the really early days).

[stuf...@gmail.com]

What may be the DNS feature rabbitmq relies on?

What should i test against?

Still can`t get it to work...

[Michael Klishin]

 On 14 February 2015 at 23:37:16, stuf...@gmail.com (stuf...@gmail.com) wrote:
> What may be the DNS feature rabbitmq relies on?

Hostname resolution for the node itself plus any nodes it (or well, epmd) is aware of.

I highly recommend strace-ing the process instead of guessing. 

[Ali Kalantari]

I have the same problem,

Rabbitmq server and the workers are on the same server and there shouldn't be a DNS resolution problem.

have you found the problem and the solution to it?

[Guillermo Prado Obando]

Hi, any solution for this issue.???

I have the exact problem, initial connection take always 8-9 seconds for different connections.

My environment:

Virtual Machine Centos 7 under KVM

My temporal solution was it install RabbitMQ under centos 6.8 and all works fine.

I think the problem, is Centos 7 and the last RabbitMQ version.

[Michael Klishin]

The problem is almost always host name resolution setup (a lookup first times out,

then a different method is tried which succeeds). There are prior discussions of this problem on this list.

Wireshark [1] is one of the go-to tools when investigating such issues.

1. https://www.rabbitmq.com/amqp-wireshark.html

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.