Increase Network Socket Limit on the RMQ Cache server

[Kishore Ponniah]

Hi All,

We have an RMQ Cache server which is serving 56 Web servers. So the cache server doesn't have enough connections to serve all of them. How to increase the network socket limit on the cache server. It is running Windows 2012 R2 Server with 8vCPU and 16GB Memory. 

Could someone suggest how to increase the network socket/how to serve all the 56 web servers?

Thanks

Kishore

[Luke Bakken]

Hi Kishore,

How do you know there aren't "enough connections"? Do you see errors logged? What are they?

How many connections do you see being made to RabbitMQ at this point?

This guide may be of use to you: https://www.rabbitmq.com/networking.html#tuning-for-large-number-of-connections

Thanks,

Luke

[Michael Klishin]

[1] explains how to adjust the open file handle limit. What makes you believe the nodes are running out of them?

Do you have evidence of inbound TCP connections from clients being rejected?

You can (and should) monitor the number of concurrent connections [2]. That would give you a better understanding of what

limit would make sense for your system. Of course you can just go with a high enough value, too.

1. https://www.rabbitmq.com/networking.html#open-file-handle-limit

2. https://www.rabbitmq.com/monitoring.html

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Kishore Ponniah]

Hi Michael/Luke,

I was talking about the Socket Descriptors because the RMQ server runs out of sockets when it has 56 web servers talking to it and refuses connections. I have attached a screenshot for your reference. 

For now, it is ok but during perf testing, RMQ servers refuse the connection.

Thanks

Kishore

[Michael Klishin]

Consider posting server logs and elaborating a bit on what evidence (facts) do you have of nodes running out of file descriptors.

About 8K descriptors may or may not be sufficient for your workload but before we suggest bumping it to 200K as the docs linked to earlier suggest, let's see some evidence.

We really prefer to not guess on this list. A single screenshot only tells so much.

[Michael Klishin]

[1] provides guidance on how to detect connection leaks, since no limit is high enough with those.

1. https://www.rabbitmq.com/connections.html#monitoring

[Kishore Ponniah]

Hi Michael,

Thanks for coming back to me on this. I do not have evidence to prove that the nodes running out of socket descriptors. The issues were raised back in Oct 2018 and whenever there is a perf testing is done the nodes will run out of sockets. I was asked to increase the socket descriptors. I have grabbed the system logs for your reference and uploaded to google drive.

https://drive.google.com/drive/folders/1U5PMeJJ4Sht9gQPKdKA9Y7B88JMM-2rb?usp=sharing

I have added ERL_MAX_PORTS to the system variables with a value of 200k and restarted the service on 1 node and I could now see the socket descriptors is increased to 200k. I have also attached the existing socket descriptors limit. 

Thanks

Kishore

[Michael Klishin]

RabbitMQ logs every inbound connection [1], so if you have logs, you can produce evidence relatively quickly.

I'm not sure what kind of response you are looking for. If you know that an extra workload runs a node over 8K descriptors, bump the limit higher and see if it helps.

Talk to whoever is responsible for running the test to see how many connections (publishers and consumers) the test uses and really needs.

The overall limit and metrics that help detect connection leaks (doc link shared above) are as much as RabbitMQ can do here.

Well, there's a per-vhost connection limit, too [2] and HAproxy or similar can help you achieve a global limit.

1. https://www.rabbitmq.com/logging.html#logged-events

2. https://www.rabbitmq.com/vhosts.html#limits

[Kishore Ponniah]

Hi Michael,

Thanks for providing more info on this issue. I have doubled the network socket limit on the cache server to 15000. Perf testing will be done tomorrow to check if the RMQ cache server is able to hold 56 web servers.

Thank you

Kishore

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitm...@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.