How to increase max connections using web-mqtt

[br...@automox.com]

* Moving this here per MK.

net.ipv4.tcp_max_syn_backlog = 65536
net.core.netdev_max_backlog = 131072
net.core.somaxconn = 4096
net.ipv4.tcp_sack = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_keepalive_intvl = 10
net.ipv4.tcp_keepalive_probes = 4
net.ipv4.tcp_keepalive_time = 60


# https://mrotaru.wordpress.com/2013/10/10/scaling-to-12-million-concurrent-connections-how-migratorydata-did-it/
net.core.wmem_default = 1024
net.core.rmem_default = 1024
net.core.rmem_max = 8192
net.core.wmem_max = 8192
net.ipv4.tcp_mem = 1024 2048 8192
net.ipv4.tcp_rmem = 1024 2048 8192
net.ipv4.tcp_wmem = 1024 2048 8192
net.ipv4.tcp_moderate_rcvbuf = 0
net.ipv4.tcp_moderate_rcvbuf = 1





[
  {rabbit, [
    {tcp_listeners, [{"0.0.0.0", 5672}]},
    {tcp_listen_options, [
                          {backlog,       4096},
                          {nodelay,       true},
                          {linger,        {true,0}},
                          {exit_on_close, false},
                          {buffer,        1024},
                          {sndbuf,        1024},
                          {recbuf,        1024}
                         ]}
  ]},

  {rabbitmq_web_mqtt, [
    {num_tcp_acceptors, 1024},
    {tcp_config, [
        {backlog,       4096},
        {nodelay,       true},
        {linger,        {true,0}},
        {exit_on_close, false},
        {buffer,        1024},
        {sndbuf,        1024},
        {recbuf,        1024}
    ]}
  ]}
].

Here is some sample client code.  Connect to any rabbit node with above 
tweaks (from Luke).  For these tests:

Ubuntu 16.04, 8 core 32G ram
Rabbitmq 3.7.4
Erlang 20.2

To run, 
go get "github.com/eclipse/paho.mqtt.golang" 
MQTT_HOST='http://127.0.0.1:15675/ws' CLIENT_COUNT=25000 go run main.go

If I set the client count to 25000, with the above tweaks and 10 
millisecond delay between clients, I only get approx. 2900 queues.

sudo rabbitmqctl list_queues | wc -l 
2936



After adjusting ranch max connections:
sudo rabbitmqctl eval 'ranch:set_max_connections(web_mqtt, 10000).'
sudo rabbitmqctl list_queues | wc -l 20001



Please let me know if there is more I need to provide.  I want to say 
thanks for all the help.

Brad

[Michael Klishin]

It could be a typo but do you really see 20K connections after you set the limit to 10K?

Web MQTT could do what RabbitMQ server already does: lift the Ranch limit entirely. We are debating

whether that should be done. Since the server does it already, I personally think it should.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[br...@automox.com]

I did see 20K. 

Some more data in my testing:

When ranch is at it's default (1024) and I try to connect 25K

sudo rabbitmqctl eval 'ranch_server:count_connections(web_mqtt).'
2044

I assume this is do to the tcp acceptor set to 1024 in advanced.config..not sure where 4 connections go =]

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[br...@automox.com]

FWIW, when I do reach around 40K connections..setting max connections to 150K, I start seeing this in the logs

2018-04-16 16:55:16.382 [info] <0.7438.20> WEB-MQTT: unexpected message {bump_credit,{<0.7458.20>,200}}
2018-04-16 16:55:16.420 [error] <0.31380.61> CRASH REPORT Process <0.31380.61> with 0 neighbours exited with reason: {timeout,{gen_server,call,[<0.31390.61>,{call,{'queue.declare',0,<<"mqtt-subscription-29ea488f-4bb0-45bb-9e60-568ff2e978e8qos0">>,false,false,false,true,false,[]},none,<0.31380.61>},60000]}} in gen_server:call/3 line 214
2018-04-16 16:55:16.420 [error] <0.31378.61> Supervisor {<0.31378.61>,rabbit_web_mqtt_connection_sup} had child cowboy_clear started with cowboy_clear:start_link(web_mqtt, #Port<0.6575087>, ranch_tcp, #{env => #{dispatch => [{'_',[],[{[<<"ws">>],[],rabbit_web_mqtt_handler,[]}]}],keepalive_sup => <0.31379.61>,...},...}) at <0.31380.61> exit with reason {timeout,{gen_server,call,[<0.31390.61>,{call,{'queue.declare',0,<<"mqtt-subscription-29ea488f-4bb0-45bb-9e60-568ff2e978e8qos0">>,false,false,false,true,false,[]},none,<0.31380.61>},60000]}} in context child_terminated
2018-04-16 16:55:16.420 [error] <0.31378.61> Supervisor {<0.31378.61>,rabbit_web_mqtt_connection_sup} had child cowboy_clear started with cowboy_clear:start_link(web_mqtt, #Port<0.6575087>, ranch_tcp, #{env => #{dispatch => [{'_',[],[{[<<"ws">>],[],rabbit_web_mqtt_handler,[]}]}],keepalive_sup => <0.31379.61>,...},...}) at <0.31380.61> exit with reason reached_max_restart_intensity in context shutdown

Thanks
Brad

[Michael Klishin]

That's not related to the number of connections per se. My guess is that a rapid build-up in client connections

(e.g. a test that uses a tight loop) results in internal database contention and thus timeouts that Web STOMP doesn't expect.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[br...@automox.com]

yeah, that makes sense.

[Brad Smith]

Saurabh Wadhwa

I am posting my changes here to take off issue list in github.

I still hit that limit even though the upgrade should not count against the 1024 limit.

For web stomp i just added this to rabbitmq_web_stomp config

 {tcp_config, [
              {backlog,       4096},
              {max_connections, infinity},
              {nodelay,       true}
          ]}

Hope that helps

Brad

[Michael Klishin]

Thanks to Brad's example we have identified what the issue is here: the default max_connections

settings in the TCP acceptor library we use is 1024 and while other protocols override it to "infinity", Web MQTT

and Web STOMP did not do that.

We've filed https://github.com/rabbitmq/rabbitmq-web-mqtt/issues/55 and https://github.com/rabbitmq/rabbitmq-web-stomp/issues/113

to make the default identical for all plugins ("no limit").

[Brad Smith]

Awesome!

Thanks for taking a look. I appreciate your time.

Brad