Connection memory consumption for Web STOMP 1.1 connection

[Mike Henders]

We have a single RabbitMQ 3.7.19 server supporting push notifications for a multi-tenant application with approximately 10,000 concurrent browser connections via Web STOMP 1.1 protocol over TLS. Each of the ~1500 tenants has its own exchange, each connection/channel has its own queue connected to that tenant's exchange.

Exchanges: 1,500 (1/tenant)

Connections/channels: 10,000

Queues: 10,000 (1/channel)

Number of queues per exchange varies between 2 and 30 depending on the tenant.

In general each of the exchanges see low message rates (a few messages per minute) with an average 0 undelivered messages in memory.

rabbitmqctl status reports the following:

{memory,
     [{connection_readers,2659272},
      {connection_writers,127596},
      {connection_channels,9060284},
      {connection_other,28310928532},
      {queue_procs,232631368},
      {queue_slave_procs,0},
      {plugins,474555452},
      {other_proc,3697837332},
      {metrics,58259844},
      {mgmt_db,544100032},
      {mnesia,29654576},
      {other_ets,4010633432},
      {binary,962478752},
      {msg_index,58784},
      {code,28840328},
      {atom,1476769},
      {other_system,4252843063},
      {allocated_unused,11906685432},
      {reserved_unallocated,0},
      {strategy,rss},
      {total,
          [{erlang,42616145416},{rss,35477737472},{allocated,54522830848}]}]},

We've read the monitoring and tuning guides on the website. We understand that RabbitMQ with a high number of connections requires a lot of memory. However, we see an average increase in connection_other memory consumption of approximately 2.7MB/connection.  That seems like a lot for a WebSocket connection that usually has 0 queued messages, heart beats every 20 seconds, and only delivers a few messages per minute.

We were hoping an upgrade from 3.6.10 to 3.7.19 would improve memory consumption but it only improved it slightly.

Is 2.7MB/connection a typical value? Anything we can do to reduce it?

Cheers,

Mike

[Mike Henders]

We did some further investigation and found that it is the {cowboy_tls,connection_process,5} processes listed by rabbitmq_top plugin which are consuming most of the memory.  What we don't understand is why many processes consume 12MiB of memory and others consume as little as 600KB of memory.

We need to be able to relate cowboy_tls processes to specific connections so we can look for patterns. Are the higher memory connections older? Do they relate to specific queues, exchanges or clients? rabbitmq_top shows links to other processes but we can't find one that tells me cowboy_tls process <0.1141.21> relates to a connection to <Remote IP Address>:<Port>.

Any suggestions on how to find this information via the web console or management API?

[Luke Bakken]

Hi Mike,

There's a chance that the hibernate_after ssl option will help in this case: http://erlang.org/doc/man/ssl.html#type-hibernate_after

Let me know if you need assistance in applying that setting in your environment. If you do, please attach all of your complete configuration files to your response.

Thanks,

Luke