Limits for rabbitmq_web_mqtt

[Johan Rhodin]

Hi team,

What are the limiting factors for number of connections with MQTT-over-WebSockets? I seem to get stuck very close to 1000 connections / node.
RabbitMQ (3.6.10 with Erlang 19.3) reports that 241548 file descriptors are available.

rabbitmq.config:
..
{rabbitmq_web_mqtt, [
{cowboy_opts, [
{max_keepalive, 10000},
{timeout, 25000}
]},
{tcp_config, [
{backlog, 8196}
]}
..


Logs contain a large number of:
"MQTT vhost picked using plugin configuration or default"

and a smaller number of:
"closing WEB-MQTT connection "127.0.0.1:38615 -> 127.0.0.1:15675" (keepalive timeout)”

/Johan

[Michael Klishin]

Should be no different from other protocols:

 * File descriptors

 * RAM (TCP buffer size)

 * Ranch acceptor count if there are connection surges

 * Inbound TCP connection backlog if there are connection surges

  * Ephemeral port range (if all connections in a test are from the same host)

and so on.

There are Cowboy options related to keepalives and connection activity timeouts:

https://ninenines.eu/docs/en/cowboy/1.0/manual/cowboy_protocol/

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send an email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[br...@automox.com]

I know this is almost a year old, but did you ever figure this out?  We can only get about 1K connections as well.   Pretty much same settings and plenty of file descriptors.

Thanks
Brad

[Luke Bakken]

Hi Brad,

I just responded to the issue you opened: https://github.com/rabbitmq/rabbitmq-web-mqtt/issues/28

Could you provide the information I requested and start a new message thread here? Thanks.

Luke

[br...@automox.com]

Luke,

Thanks for the response!

Unfortunately, the logs look normal. I turned on debug, but only see lager complaining about dropped messages.  The client outputs these errors once this wall is hit.
 
Error connecting to MQTT host: Network Error : websocket.Dial ws://172.31.20.95:4443/ws: read tcp 172.31.16.32:53584->172.31.20.95:4443: read: connection reset by peer

My test client spins up 2K connections with a keepalive (MQTT) of 30 seconds.  I have a 200 ms delay between connections.

Once the test is complete, I check the connections via rabbitmqctl list_connections.  There is always roughly around 1050-1324--This depends on tweaking rabbitmq.conf num tcp acceptors. 

I tried the same test using MQTT (without web-mqtt) and I can reach the 2k without issues. 

I am happy to give more information if needed.

Thanks again for the help!

[Luke Bakken]

Hi Brad,

I should have asked, but could you please provide the version of RabbitMQ and Erlang you are using.

Can you provide your test client code? I have recently been testing using a golang client that opens 61000+ AMQP connections on my workstation. It requires using these network-related sysctl values:

net.ipv4.tcp_max_syn_backlog = 65536

net.core.netdev_max_backlog = 131072

net.core.somaxconn = 4096

net.ipv4.tcp_sack = 1

net.ipv4.tcp_window_scaling = 1

net.ipv4.tcp_fin_timeout = 15

net.ipv4.tcp_keepalive_intvl = 30

net.ipv4.tcp_tw_reuse = 1

net.ipv4.ip_local_port_range = 1024 65535

net.ipv4.tcp_keepalive_intvl = 10

net.ipv4.tcp_keepalive_probes = 4

net.ipv4.tcp_keepalive_time = 60

# https://mrotaru.wordpress.com/2013/10/10/scaling-to-12-million-concurrent-connections-how-migratorydata-did-it/

net.core.wmem_default = 1024

net.core.rmem_default = 1024

net.core.rmem_max = 8192

net.core.wmem_max = 8192

net.ipv4.tcp_mem = 1024 2048 8192

net.ipv4.tcp_rmem = 1024 2048 8192

net.ipv4.tcp_wmem = 1024 2048 8192

net.ipv4.tcp_moderate_rcvbuf = 0

net.ipv4.tcp_moderate_rcvbuf = 1

In addition to the above, try using the following RabbitMQ configuration. Note that it is in the "old style" format that should be saved to /etc/rabbitmq/advanced.config - the settings will be merged with those in /etc/rabbitmq/rabbitmq.conf. Remove or comment out the *.num_acceptors.tcp lines in rabbitmq.conf, or change their values to 1024 as well -

[

  {rabbit, [

    {tcp_listeners, [{"0.0.0.0", 5672}]},

    {tcp_listen_options, [

                          {backlog,       4096},

                          {nodelay,       true},

                          {linger,        {true,0}},

                          {exit_on_close, false},

                          {buffer,        1024},

                          {sndbuf,        1024},

                          {recbuf,        1024}

                         ]}

  ]},

  {rabbitmq_web_mqtt, [

    {num_tcp_acceptors, 1024},

    {tcp_config, [

        {backlog,       4096},

        {nodelay,       true},

        {linger,        {true,0}},

        {exit_on_close, false},

        {buffer,        1024},

        {sndbuf,        1024},

        {recbuf,        1024}

    ]}

  ]}

].

If those settings resolve your issue, let me know. You would then want to test with higher TCP buffer sizes for both RabbitMQ and in the sysctl values.

If the settings don't resolve your issue, please provide your test code and instructions for running it. I may be able to reproduce what you are seeing locally.

Thanks,

Luke

[br...@automox.com]

FWIW,
sudo rabbitmqctl eval 'ranch:set_max_connections(web_mqtt, 100000).'

allows the clients to reach 2K.  Would you be open for a PR on exposing this through config?

[br...@automox.com]

Luke,
I am using erlang 20.2 and rabbitmq 3.7.4.

I set the files as instructed.  I was able to achieve 2048 connections of the 2100 I was trying with.  Not sure where 52 went missing.  I tried to connect 3000, but only 2048 actually connected.

Also, I see you mentioned 61000+ AMQP connections.  In my case, I am pretty sure it is the web socket plugin as I do not have this issue when connecting to port 1883. 

I did notice the plugin does not set ranch max connections.  It's default is 1024. https://ninenines.eu/docs/en/ranch/1.4/guide/listeners/

There is another knob to turn.  Without any changes to sysctl I was able to hit 2K connections with setting ranch's max.

As for the client code, It is just a simple go app based off of this https://eclipse.googlesource.com/paho/org.eclipse.paho.mqtt.golang/+/master/samples/simple.go

I do not disconnect as these clients are supposed to be long running.  I basically iterate 2K times and spawn a client like the code above and then wait.



Thanks,
Brad

[Michael Klishin]

There certainly are known WebSockets users who sustain concurrent client connection
numbers well above the 2048 Ranch limit you are mentioning.

I'd like to hear Loïc Hoguin (the author of Ranch and the Web MQTT plugin as well)'s opinion here.

Having a setting for this in RabbitMQ seems unnecessary to me since RabbitMQ core/AMQP 0-9-1 implementation

doesn't use it:

https://github.com/search?q=org%3Arabbitmq+set_max_connections&type=Code

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[br...@automox.com]

MK,
yeah the test results seemed really low to me.  I figured it was something I am doing wrong.  What trips me up is the 'expected' behavior when just hitting MQTT without web sockets. 
I would love to hear anyone's opinion on a way to do this.  Luke's config changes made it better, but not where it would be feasible to use in production.

Thanks for the response,
Brad

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

We will take a look at what may be different. So far I cannot see anythin=
g that would
=5Bforget to=5D set the limit in either the MQTT or Web MQTT plugin.

We are aware of at least one MQTT client (Paho Java) which has *client si=
de* limits
on message rates and such, which is very counterintuitive to most users.

Can you please put together a small GitHub repo we can use to reproduce t=
he 2048 connection scenario=3F
Use of Vagrant or Docker is encouraged if necessary :)

On 11 April 2018 at 05:37:38, brad=40automox.com (brad=40automox.com) wro=
te:
> MK,
> yeah the test results seemed really low to me. I figured it was somethi=
ng
> I am doing wrong. What trips me up is the 'expected' behavior when just=

> hitting MQTT without web sockets.

> I would love to hear anyone's opinion on a way to do this. Luke's confi=

g
> changes made it better, but not where it would be feasible to use in
> production.

> =20

> Thanks for the response,
> Brad
> On Tuesday, April 10, 2018 at 8:28:48 PM UTC-6, Michael Klishin wrote:
> >

> > There certainly are known WebSockets users who sustain concurrent cli=

ent
> > connection
> > numbers well above the 2048 Ranch limit you are mentioning.
> >

> > I'd like to hear Lo=C3=AFc Hoguin (the author of Ranch and the Web MQ=

TT plugin
> > as well)'s opinion here.
> > Having a setting for this in RabbitMQ seems unnecessary to me since
> > RabbitMQ core/AMQP 0-9-1 implementation
> > doesn't use it:
> >

> > https://github.com/search=3Fq=3Dorg%3Arabbitmq+set=5Fmax=5Fconnection=
s&type=3DCode =20

> >
> >
> > On Wed, Apr 11, 2018 at 5:23 AM, > wrote:
> >
> >> Luke,
> >> I am using erlang 20.2 and rabbitmq 3.7.4.
> >>

> >> I set the files as instructed. I was able to achieve 2048 connection=
s of
> >> the 2100 I was trying with. Not sure where 52 went missing. I tried =

to
> >> connect 3000, but only 2048 actually connected.
> >>
> >> Also, I see you mentioned 61000+ AMQP connections. In my case, I am

> >> pretty sure it is the web socket plugin as I do not have this issue =

when
> >> connecting to port 1883.
> >>

> >> I did notice the plugin does not set ranch max connections. It's def=

ault
> >> is 1024. https://ninenines.eu/docs/en/ranch/1.4/guide/listeners/
> >>

> >> There is another knob to turn. Without any changes to sysctl I was a=

ble
> >> to hit 2K connections with setting ranch's max.
> >>

> >> As for the client code, It is just a simple go app based off of this=

> >> https://eclipse.googlesource.com/paho/org.eclipse.paho.mqtt.golang/+=
/master/samples/simple.go =20
> >>
> >> I do not disconnect as these clients are supposed to be long running=
. I
> >> basically iterate 2K times and spawn a client like the code above an=

d then
> >> wait.
> >>
> >>
> >>
> >> Thanks,
> >> Brad
> >>

> >> On Tuesday, April 10, 2018 at 6:17:28 PM UTC-6, br...=40automox.com =
wrote:
> >>>
> >>> =46WIW,
> >>> sudo rabbitmqctl eval 'ranch:set=5Fmax=5Fconnections(web=5Fmqtt, 10=
0000).'
> >>>
> >>> allows the clients to reach 2K. Would you be open for a PR on expos=
ing
> >>> this through config=3F
> >>>
> >>> On Tuesday, April 10, 2018 at 5:40:04 PM UTC-6, br...=40automox.com=
wrote:
> >>>>
> >>>> Luke,
> >>>>
> >>>> Thanks for the response=21
> >>>>
> >>>> Unfortunately, the logs look normal. I turned on debug, but only s=
ee
> >>>> lager complaining about dropped messages. The client outputs these=

errors
> >>>> once this wall is hit.
> >>>>

> >>>> * Error connecting to MQTT host: Network Error : websocket.Dial

> >>>> ws://172.31.20.95:4443/ws : read tcp

> >>>> 172.31.16.32:53584->172.31.20.95:4443 : read: =20
> >>>> connection reset by peer*
> >>>>
> >>>> My test client spins up 2K connections with a keepalive (MQTT) of =

30
> >>>> seconds. I have a 200 ms delay between connections.
> >>>>

> >>>> Once the test is complete, I check the connections via rabbitmqctl=

> >>>> list=5Fconnections. There is always roughly around 1050-1324--This=

depends
> >>>> on tweaking rabbitmq.conf num tcp acceptors.
> >>>>

> >>>> I tried the same test using MQTT (without web-mqtt) and I can reac=

h the
> >>>> 2k without issues.
> >>>>
> >>>> I am happy to give more information if needed.
> >>>>

> >>>> Thanks again for the help=21
> >>>> On Tuesday, April 10, 2018 at 5:12:24 PM UTC-6, Luke Bakken wrote:=

> >>>>>
> >>>>> Hi Brad,
> >>>>>
> >>>>> I just responded to the issue you opened:
> >>>>> https://github.com/rabbitmq/rabbitmq-web-mqtt/issues/28
> >>>>>

> >>>>> Could you provide the information I requested and start a new mes=
sage
> >>>>> thread here=3F Thanks.

> >>>>>
> >>>>> Luke
> >>>>>
> >>>>> On Tuesday, April 10, 2018 at 3:37:33 PM UTC-7, wrote:
> >>>>>>

> >>>>>> I know this is almost a year old, but did you ever figure this o=
ut=3F
> >>>>>> We can only get about 1K connections as well. Pretty much same s=

ettings
> >>>>>> and plenty of file descriptors.
> >>>>>>
> >>>>>> Thanks
> >>>>>> Brad
> >>>>>>

> >>>>>> On Thursday, July 27, 2017 at 6:54:13 PM UTC-6, Johan Rhodin wro=
te:

> >>>>>>>
> >>>>>>> Hi team,
> >>>>>>>
> >>>>>>> What are the limiting factors for number of connections with

> >>>>>>> MQTT-over-WebSockets=3F I seem to get stuck very close to 1000 =

connections /
> >>>>>>> node.
> >>>>>>> RabbitMQ (3.6.10 with Erlang 19.3) reports that 241548 file
> >>>>>>> descriptors are available.
> >>>>>>>
> >>>>>>> rabbitmq.config:
> >>>>>>> ..

> >>>>>>> =7Brabbitmq=5Fweb=5Fmqtt, =5B
> >>>>>>> =7Bcowboy=5Fopts, =5B
> >>>>>>> =7Bmax=5Fkeepalive, 10000=7D,
> >>>>>>> =7Btimeout, 25000=7D
> >>>>>>> =5D=7D,
> >>>>>>> =7Btcp=5Fconfig, =5B
> >>>>>>> =7Bbacklog, 8196=7D
> >>>>>>> =5D=7D

> >>>>>>> ..
> >>>>>>>
> >>>>>>>
> >>>>>>> Logs contain a large number of:

> >>>>>>> =22MQTT vhost picked using plugin configuration or default=22

> >>>>>>>
> >>>>>>> and a smaller number of:

> >>>>>>> =22closing WEB-MQTT connection =22127.0.0.1:38615 -> 127.0.0.1:=
15675=22
> >>>>>>> (keepalive timeout)=E2=80=9D
> >>>>>>>
> >>>>>>> /Johan
> >>>>>>
> >>>>>> --
> >> You received this message because you are subscribed to the Google G=
roups
> >> =22rabbitmq-users=22 group.
> >> To unsubscribe from this group and stop receiving emails from it, se=
nd an
> >> email to rabbitmq-user...=40googlegroups.com <>.
> >> To post to this group, send email to rabbitm...=40googlegroups.com
> >> <>.
> >> =46or more options, visit https://groups.google.com/d/optout.

> >>
> >
> >
> >
> > --
> > MK
> >
> > Staff Software Engineer, Pivotal/RabbitMQ
> >

> =20
> --
> You received this message because you are subscribed to the Google Grou=
ps =22rabbitmq-users=22 =20
> group.
> To unsubscribe from this group and stop receiving emails from it, send =
an email to rabbitmq-users+unsubscribe=40googlegroups.com. =20
> To post to this group, send an email to rabbitmq-users=40googlegroups.c=
om.
> =46or more options, visit https://groups.google.com/d/optout.
> =20

-- =20
MK =20

Staff Software Engineer, Pivotal/RabbitMQ =20

[br...@automox.com]

MK,
I will put something together with the client test code.

thanks

[br...@automox.com]

I am seeing this in the logs when trying 75K connections.

Supervisor {<0.141.30>,rabbit_web_mqtt_connection_sup} had child cowboy_clear started with cowboy_clear:start_link(web_mqtt, #Port<0.3439747>, ranch_tcp, #{env => #{dispatch => [{'_',[],[{[<<"ws">>],[],rabbit_web_mqtt_handler,[]}]}],keepalive_sup => <0.142.30>,...},...}) at <0.143.30> exit with reason {timeout,{gen_server,call,[<0.249.30>,connect,60000]}} in context child_terminated

[br...@automox.com]

Here is some sample client code.  Connect to any rabbit node with above tweaks.  

to run,

MQTT_HOST='http://127.0.0.1:15675/ws' CLIENT_COUNT=25000 go run main.go

* You will have to do a

go get "github.com/eclipse/paho.mqtt.golang"

If I set the client count to 25000, with the above tweaks and 10 millisecond delay between clients, I only get approx. 2900 queues.

sudo rabbitmqctl list_queues | wc -l
2936
sudo rabbitmqctl list_queues | wc -l
2936
sudo rabbitmqctl list_queues | wc -l
2936

After setting ranch max connections:

sudo rabbitmqctl eval 'ranch:set_max_connections(web_mqtt, 10000).'

sudo rabbitmqctl list_queues | wc -l
20001

Please let me know if there is more I need to provide.  I want to say thanks for all the help.

Brad

[Michael Klishin]

I'd appreciate if a new thread was started with this post with the reproduction example.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brad Smith]

Will do.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.