RabbitMQ/stomp/SockJS: when are connections supposed to be closed??

[Heather Anderson]

I'm (re)visiting RabbitMQ after a few years, very glad that the setup is so much cleaner now (and I haven't needed to recompile anything yet!).

I managed to get a basic connection running with rabbitmq_web_stomp, however i'm concerned that none of the connections displayed in the management console appear to ever go away.

I did fix a bug in the client code yesterday where I didn't realize I needed to explicitly disable all heartbeats.  However I tried to do some testing today and kept getting ERR_CONNECTION_REFUSED messages until I manually restarted the service.  The management console showed around 1200 connections (all from me) over the last few days.  ProcessExplorer showed maybe 100 CLOSE_WAIT connections, but no ESTABLISHED ones.

I would like to know what I can do to start pruning dead connections.  I did find some options like tcp_keepalive_timeout that may or may not apply to STOMP connections.

I saw a message in this group earlier about someone that wrote a script that auto-prunes connections that don't correspond to any known network socket (anyone have a copy of that script?)

I'm running with a pretty simple config so far:

rabbitmq.config:

[

{rabbitmq_web_stomp, [

{tcp_config, [ %% This stanza ignored until the next release of RabbitMQ.  Sticking with https:// until then

{ip,         {10,0,114,150} },

{port,       80}

]},

{ssl_config, [

{ip,         {10,0,114,150} },

{port,       443},

{certfile,   "E:\\comet\\RabbitMQ\\private\\cert.pem"},

{keyfile,    "E:\\comet\\RabbitMQ\\private\\key.pem"},

{cacertfile, "E:\\comet\\RabbitMQ\\private\\chain.pem"},

{password,   "somePassword"}

]}

]}

].

enabled_plugins:

[rabbitmq_management,rabbitmq_web_stomp].

[Michael Klishin]

On 18 November 2015 at 01:31:54, Heather Anderson (eri...@teamworkgroup.com) wrote:
> I would like to know what I can do to start pruning dead connections.
> I did find some options like tcp_keepalive_timeout that may
> or may not apply to STOMP connections.

See http://www.rabbitmq.com/networking.html and http://www.rabbitmq.com/heartbeats.html. STOMP
does support heartbeats and in 3.6.0, Web STOMP will as well:
https://github.com/rabbitmq/rabbitmq-web-stomp/issues/15

however, in case of browser clients that are usually gone, low TCP keepalive values (defaults are
just insane for 2015) should help.

Finally, there’s an HTTP API that can close connections and window.onunload event in the browser. 
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Heather Anderson]

I will study this a bit (thank you).

A quick note, if you do add support for heartbeats for "all clients except SockJS", can you please report heat-beat:0,0 in the CONNECT frame when SockJS is being used?  The main reason I'm force-disabling heartbeats right now is that both client and server may report support but no heartbeats are ever received (so the browser creates a new connection every 30sec, which RabbitMQ never closes)

[Michael Klishin]

On 18 November 2015 at 03:20:01, Heather Anderson (eri...@teamworkgroup.com) wrote:
> A quick note, if you do add support for heartbeats for "all clients
> except SockJS", can you please report heat-beat:0,0 in the CONNECT
> frame when SockJS is being used? The main reason I'm force-disabling
> heartbeats right now is that both client and server may report
> support but no heartbeats are ever received (so the browser creates
> a new connection every 30sec, which RabbitMQ never closes)

Can you please file an issue for web-stomp on GitHub from the above?

Thank you. 

[Heather Anderson]

Can I get a good idea of what the focus of the issue would be?  That heartbeats aren't working, the server is misreporting heartbeat support, or that the server is (for me) not dropping connections without working heartbeats?

According to RabbitMQ documentation heartbeats with stomp are impossible, further searches indicate both STOMP and SockJS have their own heartbeats, either one of which can detect a session closure.

Current tests show the server maxes out at 1123 connections/channels/queues/consumers, further connections are currently timing out.

My inexperience with Erlang is frustrating, the closest I've gotten to a "responsible party" is seeing the word "tcp_closed" appearing in sockjs_action:reply_loop and wondering what the heck the apparent return value means (and where that value ends up being sent to).

Thank you for your responses, this is a lot more meaningful (and faster) than I was ever expecting from a user group.

[Michael Klishin]

On 18 November 2015 at 10:43:07, Heather Anderson (eri...@teamworkgroup.com) wrote:
> > Can I get a good idea of what the focus of the issue would be? That
> heartbeats aren't working, the server is misreporting heartbeat
> support, or that the server is (for me) not dropping connections
> without working heartbeats?

That for SockJS heartbeats should be reported as 0, 0 (if that’s possible at all).

> According to RabbitMQ documentation heartbeats with stomp
> are impossible, further searches indicate both STOMP and SockJS
> have their own heartbeats, either one of which can detect a session
> closure.

See the issue I’ve linked to: the changes are specific to master so they cannot be reflected
in the live docs. 

[Heather Anderson]

Okay, I've posted the requested issue.  The state of things on my side as far as I can see:

(*) STOMP heartbeats over SockJS are impossible as stated on https://www.rabbitmq.com/web-stomp.html.  I haven't found an explanation why, but with the greater experience with SockJS the developers must have I'm guessing the reason has to be good (and frustrating).

(*) SockJS heartbeats as default implemented are one-way and the client is explicitly required to ignore them.  SockJS added explicit bidirectional heartbeats in issues #64 / #67 / #69 / #167, however none of these appear to have filtered their way to either the SockJS or RabbitMQ branches of sockjs-erlang.

(*) As described in SockJS issues #64, when using the dual-channel SockJS protocol tcp settings alone will not prevent the channel from staying open indefinitely (or at least until the client machine starts explicitly RSTing the packets).  I've got Fiddler2 open right now a couple hours after closing a test window and it shows half a dozen active xhr_streaming connections.

My previous foray into this (using orbited back then rather than SockJS) ended rather nastily when connections would go silent a few minutes into the page load (and for unknown reason could not be re-established without a full page refresh).  Things do seem more stable now and I may just need to hold my breath a couple more years (?!?) until our minimum-supported client implements some form of raw websockets that SockJS can polyfill into.  Even with a fully-stable message transport, I do have some concern that someone could spew a bunch of SockJS CONNECT frames to the RabbitMQ server and apparently saturate the server with limited auto-recycling available.