Too many open files (due to ws) / lsof issues with sockjs-node

[Marek Majkowski]

Guys,

@yurynix on github has an interesting suggestion regarding
the stuck websocket connections on sockjs-node.

The possible culprit is faye-websocket-node which seems
to be stuck in API.CLOSING state for way too long sometimes.

Take a look at this comment:
https://github.com/sockjs/sockjs-node/issues/99#issuecomment-11084738

Can anyone verify if, and in what circumstances, websocket
connections get stuck in API.CLOSING state on faye-websocket-node
layer?

Can anyone verify if a patch like that to sockjs-node
(setting ack=false) does solve the problem?

$ git diff src/trans-websocket.coffee
diff --git a/src/trans-websocket.coffee b/src/trans-websocket.coffee
index 5c07344..036485c 100644
--- a/src/trans-websocket.coffee
+++ b/src/trans-websocket.coffee
@@ -97,7 +97,7 @@ class WebSocketReceiver extends transport.GenericReceiver
didClose: ->
super
try
- @ws.close()
+ @ws.close(1001, "error", false)
catch x
@ws = null
@connection = null
@@ -136,7 +136,7 @@ class RawWebsocketSessionReceiver extends transport.Session
if @readyState isnt Transport.OPEN
return false
@readyState = Transport.CLOSING
- @ws.close(status, reason)
+ @ws.close(status, reason, false)
return true

didClose: ->
@@ -145,7 +145,7 @@ class RawWebsocketSessionReceiver extends transport.Session
@ws.removeEventListener('message', @_message_cb)
@ws.removeEventListener('close', @_end_cb)
try
- @ws.close()
+ @ws.close(1001, "error", false)
catch x
@ws = null

[Shripad K]

On Thu, Dec 6, 2012 at 6:50 PM, Marek Majkowski <maj...@gmail.com> wrote:

Guys,

@yurynix on github has an interesting suggestion regarding
the stuck websocket connections on sockjs-node.

The possible culprit is faye-websocket-node which seems
to be stuck in API.CLOSING state for way too long sometimes.

Take a look at this comment:
  https://github.com/sockjs/sockjs-node/issues/99#issuecomment-11084738

Can anyone verify if, and in what circumstances, websocket
connections get stuck in API.CLOSING state on faye-websocket-node
layer?

I have given my reasoning here: https://github.com/sockjs/sockjs-node/issues/99#issuecomment-11093966

[Shashwat Srivastava]

Hi Marek,

Sorry for not being able to reply earlier (in the past thread).

I am using the fix suggested by you and it seems to be working so far. Thank you so much! Will have to keep running the server for few days to be sure.

netstat shows the following:

FIN_WAIT2 6

CLOSING 1

LISTEN 4

CLOSE_WAIT 11

TIME_WAIT 726

ESTABLISHED 1301

LAST_ACK 2

FIN_WAIT1 3

Thanks again,

Shashwat

[Shashwat Srivastava]

Also, lsof -p <PROCESS_ID_OF_SERVER> | grep identify | wc gives following output: '0       0       0'   :)

Thanks,

Shashwat

[theyak]

Woot! Things look good now! Thanks!

[Shashwat Srivastava]

The memory being used by sockjs-node seems to increase continuously when websocket transport is used. Although lsof (file descriptor) issue no longer seems to exist. Any idea, why is this happening?

Thanks,

Shashwat

[Marek Majkowski]

On Sat, Dec 8, 2012 at 2:23 PM, Shashwat Srivastava <dar...@gmail.com> wrote:
> The memory being used by sockjs-node seems to increase continuously when
> websocket transport is used. Although lsof (file descriptor) issue no longer
> seems to exist. Any idea, why is this happening?

No, but feel encouraged to investigate.

Marek

[Shashwat Srivastava]

Thanks. I am trying.

[Shashwat Srivastava]

theyak, are you also experiencing this?

[theyak]

Been running my test server, used in conjunction with a live site, for a couple days now. I have not noticed any memory leaks, unless they are very slow leaks, which is possible. I'd like to know more about how Shashwat is seeing the memory leaks to maybe try to replicate and/or confirm the leak.

[Shashwat Srivastava]

I have noticed so far that when I enable ws transport, memory usage almost seems to be constant if number of connections is constant. When the number of connections changes rapidly (i.e. it goes from low to high), then memory usage also increases (which should happen), but it doesn't fall back when connections fall down. And, hence this cycle is repeated again and again, and RAM keeps on building gradually. So, this seems to happen when ws connections are established and destroyed rapidly. I was hoping that memory usage would reach some kind of equilibrium eventually but this didn't happen.

[theyak]

I still have not seen any memory issues. However, I have this on my node server's socket.on("close"):

socket.on( "close", function() {
    socket.removeAllListeners();

        .
        .
        .
});

First, I'm not sure if I should have the removeAllListeners there. I added it during tests to try to get things to work better prior to 0.3.5. Is it safe to have it? Is it possible that without it a memory leak can occur?

[Shashwat Srivastava]

My node.js version is 0.8.15. I am also using sockjs 0.3.5. Earlier I was using faye-websocket 0.4.0. I have tried with both 0.4.3 and master branches. Additionally, I tried the code change in api.js of faye-websocket as discussed here - https://github.com/sockjs/sockjs-node/issues/99. For me, memory leak is always there when I enable ws transport. What is your node.js version and which faye-websocket lib are you using?

I am not using removeAllListeners. I am not sure if it should be there. I will give it a try.

Thanks,

Shashwat