RabbitMQ 3.5.0 is very sluggish on my MacBookPro laptop

[vitaly numenta]

Ever since upgrading to RabbitMQ v3.5.0, interaction with RabbitMQ on my MacBookPro has become incredibly slow most of the time. Occasionally, after restarting the machine, it gets fast for a bit, and then becomes sluggish again. Something as simple as establishing an AMQP connection (pika.BlockingConnection) takes about 8 seconds, and refreshing the Overview tab view in RabbitMQ web gui takes about 15 seconds to complete.

I've tried cleared browsing data and resetting rabbitmq via `rabbitmqctl stop_app; rabbitmqctl reset; rabbitmqctl start_app`, but nothing seems to help.

System: OS X Yosemite 10.10.3 with all the latest updates.

Hardware Overview:

  Model Name: MacBook Pro

  Model Identifier: MacBookPro10,1

  Processor Name: Intel Core i7

  Processor Speed: 2.7 GHz

  Number of Processors: 1

  Total Number of Cores: 4

  L2 Cache (per Core): 256 KB

  L3 Cache: 6 MB

  Memory: 16 GB

Storage: mostly free 500GB SSD

$ rabbitmqctl status

Status of node rabbit@localhost ...

[{pid,1836},

 {running_applications,

     [{rabbitmq_management_visualiser,"RabbitMQ Visualiser","3.5.0"},

      {rabbitmq_management,"RabbitMQ Management Console","3.5.0"},

      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.5.0"},

      {webmachine,"webmachine","1.10.3-rmq3.5.0-gite9359c7"},

      {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.5.0-git680dba8"},

      {rabbitmq_mqtt,"RabbitMQ MQTT Adapter","3.5.0"},

      {rabbitmq_stomp,"Embedded Rabbit Stomp Adapter","3.5.0"},

      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.5.0"},

      {rabbitmq_amqp1_0,"AMQP 1.0 support for RabbitMQ","3.5.0"},

      {rabbit,"RabbitMQ","3.5.0"},

      {mnesia,"MNESIA  CXC 138 12","4.12.1"},

      {os_mon,"CPO  CXC 138 46","2.2.15"},

      {amqp_client,"RabbitMQ AMQP Client","3.5.0"},

      {inets,"INETS  CXC 138 49","5.10.2"},

      {xmerl,"XML parser","1.3.7"},

      {sasl,"SASL  CXC 138 11","2.4"},

      {stdlib,"ERTS  CXC 138 10","2.1"},

      {kernel,"ERTS  CXC 138 10","3.0.1"}]},

 {os,{unix,darwin}},

 {erlang_version,

     "Erlang/OTP 17 [erts-6.1] [source] [64-bit] [smp:8:8] [async-threads:30] [hipe] [kernel-poll:true]\n"},

 {memory,

     [{total,47676552},

      {connection_readers,27288},

      {connection_writers,0},

      {connection_channels,0},

      {connection_other,66048},

      {queue_procs,90752},

      {queue_slave_procs,0},

      {plugins,616560},

      {other_proc,13949936},

      {mnesia,76464},

      {mgmt_db,427696},

      {msg_index,59784},

      {other_ets,1316064},

      {binary,3388264},

      {code,21051922},

      {atom,744345},

      {other_system,5861429}]},

 {alarms,[]},

 {listeners,

     [{clustering,25672,"::"},

      {amqp,5672,"127.0.0.1"},

      {stomp,61613,"::"},

      {mqtt,1883,"::"}]},

 {vm_memory_high_watermark,0.4},

 {vm_memory_limit,6323958579},

 {disk_free_limit,50000000},

 {disk_free,421833146368},

 {file_descriptors,

     [{total_limit,156},{total_used,12},{sockets_limit,138},{sockets_used,6}]},

 {processes,[{limit,1048576},{used,237}]},

 {run_queue,0},

 {uptime,9642}]

[vitaly numenta]

rab...@localhost.log from two back-to-back connection requests demonstrates the 8-second connection time:

=INFO REPORT==== 18-Apr-2015::02:56:07 ===

accepting AMQP connection <0.426.0> (127.0.0.1:54377 -> 127.0.0.1:5672)

=INFO REPORT==== 18-Apr-2015::02:56:15 ===

accepting AMQP connection <0.438.0> (127.0.0.1:54379 -> 127.0.0.1:5672)

These were the result of executing:

def f():

    c = pika.BlockingConnection()

    c2 = pika.BlockingConnection()

Everything else on the system is fast.

[vitaly numenta]

I run rabbitmq as `rabbitmq-server -detached`

[vitaly numenta]

And having the same problem with RabbitMQ v3.5.1.

Earlier version (either 3.2.x or 3.3.x) worked like a charm - fast and never any problems.

[Michael Klishin]

On 18 April 2015 at 13:17:14, vitaly numenta (vitaly.kru...@gmail.com) wrote:
> Something as simple as establishing an AMQP connection (pika.BlockingConnection)
> takes about 8 seconds, and refreshing the Overview tab view in
> RabbitMQ web gui takes about 15 seconds to complete.

Check if your machine's hostname resolves without going through DNS:
 

and Memory Pressure in the Activity Monitor
(should not affect connection times unless there's swapping but RabbitMQ uses 47 MB)

The only know regression in 3.5.0 is fixed in 3.5.1 and again, cannot affect connection time in any significant way:
https://github.com/rabbitmq/rabbitmq-server/issues/69
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

 On 18 April 2015 at 13:17:14, vitaly numenta (vitaly.kru...@gmail.com) wrote:

This also suggests a hostname resolution timeout:
https://groups.google.com/d/msg/rabbitmq-users/lxVCAi5ntHY/tKJU0MIcRQ8J

[vitaly numenta]

Hi Michael, many thanks, as always, for your insight. It turns out that the sluggish behavior occurs when VPN is turned on, which is configured for "Send all traffic over VPN connection". When VPN is turned off, performance recovers. This explains why rabbitmq performance would suddenly improve for a short time after restarting Mac OS: my VPN is configured to start manually, so it's off upon completion of restart.

However, I never noticed this sluggishness until I upgraded to rabbitmq 3.5.x, which suggests that something changed in the broker. Here are some additional bits:

When my VPN is turned on and pika.BlockingConnection() is experiencing repeatable sluggishness (the 8 seconds or so to connect), all of the following happen to execute pretty fast:

In [33]: socket.socket().connect(("vkruglikovs-MacBook-Pro.local", 80))

In [34]: socket.socket().connect(("localhost", 80))

In [35]: socket.socket().connect(("vkruglikovs-MacBook-Pro.local", 15672))

In [36]: socket.socket().connect(("localhost", 15672))

In [37]: socket.socket().connect(("localhost", 5672))

So, which host is rabbitmq trying to connect to that results in the sluggish behavior?

pika's default connection host is "localhost".

Below is the output of Chrome browser's Network tab while re-loading the Overview tab in the management GUI. You can see that "overview" consumed almost 11 seconds and "nodes" about 8 seconds:

main.css GET 304 text/css (index):18 140 B 6 ms

login.ejs?0.47063263528980315 GET 200 text/plain ejs.min.js:1 761 B 2 ms

rabbitmqlogo.png GET 304 image/png jquery-1.6.4.min.js:3 140 B 2 ms

whoami GET 200 application/json main.js:933 239 B 2 ms

layout.ejs?0.9595843909773976 GET 200 text/plain ejs.min.js:1 1.3 KB 2 ms

overview GET 200 application/json main.js:933 1.9 KB 10.77 s

nodes GET 200 application/json main.js:933 4.1 KB 8.01 s

vhosts GET 200 application/json main.js:933 740 B 3 ms

vhosts GET 200 application/json main.js:933 740 B 3 ms

extensions GET 200 application/json main.js:933 263 B 4 ms

visualiser.js GET 304 application/x-javascript main.js:148 140 B 3 ms

dispatcher.js GET 304 application/x-javascript main.js:148 140 B 3 ms

overview?lengths_age=60&lengths_incr=5&msg_rates_age=60&msg_rates_incr=5 GET 200 application/json main.js:889 6.2 KB 16.01 s

nodes GET 200 application/json main.js:889 4.1 KB 8.01 s

overview.ejs?0.3485896633937955 GET 200 text/plain ejs.min.js:1 10.0 KB 2 ms

[Michael Klishin]

VPN changes your DNS server upon connection. Just another sign of a DNS lookup timeout.

MK

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[vitaly numenta]

Hi Michael, thanks for your follow-up. It's clear that VPN has some sort of impact on RabbitMQ 3.5.x, but what's surprising is that DNS resolution for localhost using python API is *instant* under the circumstances. This includes both socket.gethostbyname and socket.getaddrinfo (ipv4 and ipv6). Same with `dscacheutil -q host -a name www.example.com` - instant!.

*** It looks like that RabbitMQ is using a questionable dns resolution method that bypasses both /etc/hosts and /private/etc/hosts on Mac OS X and going directly to the network. Otherwise it would have instantly picked up both IPv4 and IPv6 addresses from the hosts files, just like python API and dscacheutil. Both of my mac's hosts files contain localhost mappings for both IPv4 and IPv6.

/etc/hosts:

127.0.0.1 localhost

255.255.255.255 broadcasthost

::1             localhost

/private/etc/hosts:

127.0.0.1 localhost

255.255.255.255 broadcasthost

::1             localhost

[Michael Klishin]

On 19 April 2015 at 07:46:20, vitaly numenta (vitaly.kru...@gmail.com) wrote:
> It looks like that RabbitMQ is using a questionable dns resolution
> method that bypasses both /etc/hosts and /private/etc/hosts
> on Mac OS X and going directly to the network

Vitaly,

RabbitMQ 3.5 takes extra effort to ensure epmd is started and running because in some environments
system logouts or administrators may result in epmd terminating (once that happens, rabbitmqctl stops
working). There were no other changes that may involve hostname resolution I can think of, in 3.4 or 3.5.

According to Erlang documentation [1][2],
the runtime can use just DNS or an OS-provided mechanism (which is typically /etc/hosts or similar and then DNS).
It even has its own file-based mechanism for environments where /etc/hosts cannot be modified. RabbitMQ
does not use it, although it is possible to opt in by overriding VM command line flags.

The only DNS-related setting RabbitMQ has is reverse DNS lookups for client IPs:
https://github.com/rabbitmq/rabbitmq-server/blob/master/docs/rabbitmq.config.example#L41-45
which is off by default.

Since VPN likely changes your hostname,
you may want to make sure that *that new hostname* can resolve via /etc/hosts or DNS, too,
if that's the resolution mechanism you want to use. 

1. http://www.erlang.org/doc/man/epmd.html
2. http://www.erlang.org/documentation/doc-4.9.1/doc/installation_guide/corrective.html

[vitaly numenta]

Thanks Michael! I added my machine's `hostname` on both localhost lines in /etc/hosts (it appears that /private/etc/hosts is the same inode) and now rabbitmq is instant with and without VPN!

I guess I don't understand why rabbitmq is using `hostname` instead of "localhost". It thought that "localhost" would be the more natural choice.

Anyway, thanks again for your great help!

[Michael Klishin]

On 19 April 2015 at 08:39:56, vitaly numenta (vitaly.kru...@gmail.com) wrote:
> I guess I don't understand why rabbitmq is using `hostname`
> instead of "localhost". It thought that "localhost" would be
> the more natural choice.

"localhost" won't work very well once you go beyond a single machine.
RabbitMQ startup scripts cannot know if they are running on a developer's laptop or production
servers.

Node name can be overridden with an environment variable however you please.