Websocket connection drops but client doesn't recognize

[Gopal Patel]

We've been experiencing websocket connections dropping, usually when computers sleep or change network connections, but not consistently in those scenarios. The Faye JS client doesn't notice and reconnect.

Running Faye 0.8.8 on both server (ruby) and client, but this has been happening since 0.8.2. Chrome (multiple versions) on OS X on multiple machines. It's hard to tell what the socket is doing because the web inspector has never been open with the websocket item already recording frames.

If I inspect the client object on a page experiencing this:

`client._transport._state` is 3 (CONNECTED)

`client._messageId` is stuck on the last ID (doesn't increment every 25s as usual)

In some cases, `_client._transport._socket.readyState` is 3.

Sorry it's a bit vague, but it's been hard to reproduce although we get at least a couple reports a day from users.

Wondering if anyone's seen similar behavior?

Thanks,

-Gopal

[Nicholas Firth-McCoy]

Hi Gopal,

I've been looking into this lately too. From what I've turned up, it looks the like the `onclose` event on the websocket isn't firing. Apparently, this can take quite some time to happen, as Chrome won't close the websocket until the TCP connection times out and stops retrying. See http://code.google.com/p/chromium-os/issues/detail?id=14734 and http://code.google.com/p/chromium/issues/detail?id=76358 for related info. There's some disagreement in those threads about when it's appropriate to trigger `onclose`.

On my system, I changed `net.ipv4.tcp_retries2` from 15 to 1, and then turned off my network. It took about 5 minutes for `onclose` to fire, which caused `client._transport._state` to change to 1 and Faye to trigger `transport:down`. With the default setting, I believe it took over 30 minutes of network downtime for this to occur. It's time consuming to reproduce so I haven't bothered to.

You may be able to use `navigator.onLine` in conjunction with the `transport:down` event on the client to keep better track of how online you are. This will toggle as soon as your network manager state changes (eg wifi drops, network shutdown for computer to sleep etc).

That said, there may well be other issues with the `onclose` event not firing. This is just my first foray into figuring it out!  In any case, I don't expect there's anything Faye can do about `onclose` not happening, and it appears to behave correctly when it does happen.

All the best,
Nicholas

[James Coglan]

On 18 January 2013 07:02, Nicholas Firth-McCoy <n...@nfm.id.au> wrote:

That said, there may well be other issues with the `onclose` event not firing. This is just my first foray into figuring it out!  In any case, I don't expect there's anything Faye can do about `onclose` not happening, and it appears to behave correctly when it does happen.

Apologies for my delay in replying, but I'd just like to confirm this. Faye can only go by what the browser tells it about the state of a connection, and under some circumstances browsers do an unsatisfactory job here. 

[Paul]

I sort of worked around this (I'm 90% sure this is the same issue I ran into) by sending a "PING" message from the client to server every 30s (turning on ping from server to client isn't the same thing) which an extension intercepts and returns immediately with a failure that I ignore. Forcing this through will most of the time (I've seen one case where it needed to time out completely but I haven't been able to reproduce since) make it realize it's actually broken and Faye will reconnect it. 

[James Coglan]

On 4 February 2013 16:29, Paul <psel...@gmail.com> wrote:

I sort of worked around this (I'm 90% sure this is the same issue I ran into) by sending a "PING" message from the client to server every 30s

You mean a WebSocket ping? How do you make the client do this? This doesn't require server-side support, right? faye-websocket should send a PONG back automatically without the user needing to do anything.

[Paul]

A reverse of the WebSocket ping that is initiated by the client instead. If the socket has broken but somehow the client doesn't recognize it as closed yet, the server-initiated ping never gets to the client, so the client never gets a chance to respond with a pong.

I'm just doing something like this:

function ping(){

client.publish('/service/ping');

}

setInterval(ping, 60000);

I have an extension listening for publishes to /service/ping that returns immediately with an error, which the client ignores.

This seems to kickstart the "reconnect" process in some cases where the connection is somehow broken but the client doesn't know it yet.

[James Coglan]

On 4 February 2013 22:53, Paul <psel...@gmail.com> wrote:

A reverse of the WebSocket ping that is initiated by the client instead. If the socket has broken but somehow the client doesn't recognize it as closed yet, the server-initiated ping never gets to the client, so the client never gets a chance to respond with a pong.

I'm just doing something like this:

function ping(){

client.publish('/service/ping');

}

setInterval(ping, 60000);

Oh right -- this is not a WebSocket ping (meaning a PING frame within the WedSocket protocol, which I don't think there's an API for in JS), it's a Bayeux message you're sending just in order to send some data and make the socket realize it's dead. 

[Paul]

Exactly. Sorry, shouldn't have conflated the terms.

[James Coglan]

On 5 February 2013 14:22, Paul <psel...@gmail.com> wrote:

Exactly. Sorry, shouldn't have conflated the terms.

No worries :)