Faye nodejs server queries

[Neeraj Agarwal]

Hi, I am trying to use faye nodejs server, to which multiple android devices connect, and publish and subscribe to messages. I am pretty much new to nodejs and faye, and had a few queries.

1) CPU
I am trying to use the 'cluster' package to run multiple instances of the app which share the same port. How many cores should I leave for other processes? On superficial thought, only forever monitoring agent, data monitoring agent and garbage collector of javascript are the only other processes that I need to run on th system. Initially I was running 2 instances (1 master js process whcih forks children processes, 2 children processes which do the actual work) on a 2 core machine, 4 GB ram. But it got stuck at one point where 1 cpu was being utilized 100% and the other 1 was around 20-30%, but everything was stuck. I restarted the app and things became fine.

2) Memory usage:
I see that memory of the system keeps on increasing for some hours, and at some point of time it drops down. A few times it has got stuck to a high value and never got down, and I had to restart the app to reset the memory usage.
On some searching, it looks like it is a feature of javascript V8 engine, that only when memory piles up, it runs a garbage collector and brings down the memory. Could it be that since I am running 2 instances on 2 cores, GC is not actually getting any cpu time?

3) Client trying to connect

2 clients, I don't know from where, are sending ' {"channel":"/meta/connect","clientId":"xmge7z3g8nxbiu1ocyu2mpfqxk2jb1f","connectionType":"websocket","id":"2az"}' to server continuously (every millisecond it seems)

These are the faye logs for the same:

[2014-10-13 20:20:17.093] [DEBUG] faye - [Faye.NodeAdapter] Received message via WebSocket[hybi-13]: "[{\"channel\":\"/meta/connect\",\"clientId\":\"xmge7z3g8nxbiu1ocyu2mpfqxk2jb1f\",\"connectionType\":\"websocket\",\"id\":\"2az\"}]"
[2014-10-13 20:20:17.093] [INFO] faye - [Faye.Server] Processing messages: [{"channel":"/meta/connect","clientId":"xmge7z3g8nxbiu1ocyu2mpfqxk2jb1f","connectionType":"websocket","id":"2az"}] (local: false)
[2014-10-13 20:20:17.093] [DEBUG] faye - [Faye.Server] Passing through "incoming" extensions: {"channel":"/meta/connect","clientId":"xmge7z3g8nxbiu1ocyu2mpfqxk2jb1f","connectionType":"websocket","id":"2az"}
[2014-10-13 20:20:17.094] [INFO] faye - [Faye.Server] Handling message: {"channel":"/meta/connect","clientId":"xmge7z3g8nxbiu1ocyu2mpfqxk2jb1f","connectionType":"websocket","id":"2az"} (local: false)
[2014-10-13 20:20:17.094] [DEBUG] faye - [Faye.Engine] Flushing connection for "xmge7z3g8nxbiu1ocyu2mpfqxk2jb1f"
[2014-10-13 20:20:17.094] [DEBUG] faye - [Faye.Engine] Closing connection for "xmge7z3g8nxbiu1ocyu2mpfqxk2jb1f"
[2014-10-13 20:20:17.094] [DEBUG] faye - [Faye.Server] Processing reply: {"id":"2az","channel":"/meta/connect","successful":false,"error":"401:xmge7z3g8nxbiu1ocyu2mpfqxk2jb1f:Unknown client","advice":{"reconnect":"handshake"}}
[2014-10-13 20:20:17.094] [DEBUG] faye - [Faye.Server] Passing through "outgoing" extensions: {"id":"2az","channel":"/meta/connect","successful":false,"error":"401:xmge7z3g8nxbiu1ocyu2mpfqxk2jb1f:Unknown client","advice":{"reconnect":"handshake"}}
[2014-10-13 20:20:17.095] [INFO] faye - [Faye.Server] Returning replies: [{"id":"2az","channel":"/meta/connect","successful":false,"error":"401:xmge7z3g8nxbiu1ocyu2mpfqxk2jb1f:Unknown client","advice":{"reconnect":"handshake"}}]

Can somebody tell me what it means? And can I identify the ip address from this client id somehow so that I can at least identify from where these messages are coming in?

[Neeraj Agarwal]

And can I know when the garbage collector runs? and what exactly it is doing?

[James Coglan]

On 13 October 2014 16:29, Neeraj Agarwal <neeraj....@olacabs.com> wrote:

2 clients, I don't know from where, are sending ' {"channel":"/meta/connect","clientId":"xmge7z3g8nxbiu1ocyu2mpfqxk2jb1f","connectionType":"websocket","id":"2az"}' to server continuously (every millisecond it seems)

It's quite possible you're running into a well-known bug in 1.0 to do with client-side error handling. Could you try the current `master` branch and see if that stops this happening? Everyone who's been experiencing this problem has reported improvements using that codebase. 

[James Coglan]

On 13 October 2014 16:29, Neeraj Agarwal <neeraj....@olacabs.com> wrote:

2 clients, I don't know from where, are sending ' {"channel":"/meta/connect","clientId":"xmge7z3g8nxbiu1ocyu2mpfqxk2jb1f","connectionType":"websocket","id":"2az"}' to server continuously (every millisecond it seems)

[Neeraj Agarwal]

I am using npm for faye, and 1.0.3 is the latest available it seems. I tried to checkout master branch, but it is totally different from the node module that was added when i installed package 1.0.3. Even the 1.0.3 git tag is different from what I saw in node modules folder.
I tried tweaking faye_node.js code by sending success as true, the correct/wrong clientId in response, advice as 'none' instead of reconnect, but nothing seems to work. I tried adding the clientId in '/clients' zset in redis, but it looked like faye-redis removed it from there after a few seconds.
The message comes in ws.onmessage, so it seems like it is not internal to server, it is coming from some client. Can I identify the ip address of the client using the websocket object at least?
Can you help me in replacing  the faye node module with the master branch, or maybe at least suggest a tweak that could solve this temporarily until then?

[James Coglan]

On 17 October 2014 15:46, Neeraj Agarwal <neeraj....@olacabs.com> wrote:

I am using npm for faye, and 1.0.3 is the latest available it seems. I tried to checkout master branch, but it is totally different from the node module that was added when i installed package 1.0.3. Even the 1.0.3 git tag is different from what I saw in node modules folder.
I tried tweaking faye_node.js code by sending success as true, the correct/wrong clientId in response, advice as 'none' instead of reconnect, but nothing seems to work. I tried adding the clientId in '/clients' zset in redis, but it looked like faye-redis removed it from there after a few seconds.
The message comes in ws.onmessage, so it seems like it is not internal to server, it is coming from some client. Can I identify the ip address of the client using the websocket object at least?
Can you help me in replacing  the faye node module with the master branch, or maybe at least suggest a tweak that could solve this temporarily until then?

You won't have any luck making small tweaks to 1.0.3 -- fixing these bugs required a big refactoring of the Faye client. To build Faye from source, here's what you do:

$ git clone git://github.com/faye/faye.git

$ cd faye

$ npm install

$ npm run-script build

This will generate a directory called 'build', and that's what's in the npm package. 

[Neeraj Agarwal]

@James, As you suggested, I have deployed master branch on production. And the issue has gone away.
(I saw your response in one of the issues, you don't guarantee that it is production ready, but people are using it in production. And I don't see any issues being reported for it.)

[Neeraj Agarwal]

Should I use the master branch in my server side clients as well?

[James Coglan]

On 12 November 2014 08:43, Neeraj Agarwal <neeraj....@olacabs.com> wrote:

Should I use the master branch in my server side clients as well?

Yes, I would advise using the same version of Faye throughout. Although, the server and client are designed to work with *any other* Bayeux implementation, so it doesn't matter if your Faye server and client are running different versions. 

[Neeraj Agarwal]

Hi,

I am using server side faye ruby client, and many times I see that the faye server is getting messages (at a 300-400 per second rate) but they are not being sent to the client, maybe for 30-40 seconds. And then suddenly all messages are pumped to the client.
This one time I restarted faye server and looked like the client failed to reconnect on its own.

Faye version on the client (ruby) is 1.0.3, and the faye version on server is the master branch (nodejs). Does 1.0.3 ruby client has any such bug? Should I switch to master on the client too?

[Neeraj Agarwal]

Just saw, 1.1.0 is out.  And there is one more commit (seems like a bug fix) after that in the master branch. Do you suggest we should move our production server on the new faye version? Or the current master branch?
(I couldn't find changelog or release notes anywhere, so asking here)