Seeing stability issues with nodejs 0.8.x on Linux

[Hugo]

Immediately after upgrading from node.js 0.6.18 to 0.8.8 (and later to 0.8.9) I started seeing signs of instability where the server occasionally does not acknowledge an incoming request, especially under (moderate) load. The miss seems to be happening every 100 requests or so.

Has anyone seen these symptoms?

PS: just reverted back to 0.6.18 to confirm that the problem is isolated down to the version of node.js (seems to be the case so far but it will take ~24hrs for me to get a definite conclusion).

[Hugo]

Confirmed :( I have been running almost 2 weeks with node 6.18 without a glitch. With 0.8.9 I was seeing 2-3 issues per day at the minimum

Unfortunately I don't have much of a diagnosis, I found out the issue because I am monitoring the server with monit (it tests the server with a basic http request that does nothing other than acknowledging with a 200 code)

For every failure, artifacts of the monit request was missing from the logs - which leads me to believe that the server never acknowledged the request(?)

Note that the version of node is the only factor that was changed between tests (same code, same server, same config for all components)

Has anyone experience anything of a similar nature?

[Murvin Lai]

few things you may want to check.

- are you running with default 1GB memory or increase to ~2GB?  

- default or production mode?

- I usually run with this flag:  node -nouse-idle-notification -expose-gc   myHttpServer.js

- does your server make any remote service call?

and what is the CPU / memory usage during the glitch?

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

[Hugues Hardel]

Hey Thanks for the reply Murvin: answers to your config questions below

I should probably mention first that this is a small / low pressure server (a small server with 256MB RAM is plenty to handle our traffic, which realistically never amounts to more than than 2-3requests/sec when very busy)

- Running with default memory config

- We're not running with socket.io or express or any other package with production v. default configuration, so I'd have to say this probably doesn't apply (unless I am misunderstanding your question?)

- We run the node process without any system flags - as I understand them both -nouse-idle-notification & -expose-gc can be used to optimize garbage collection when GC is a problem but I don't have any reasons to think that would be the case here

- The server makes plenty of remote service calls if by service call you mean http requests to 3rd party servers. What did you have in mind there?

- Memory usage and cpu seemed normal before the glitch aka 30-40MB heap, CPU in the low single digits (note that the server is still serving requests after missing the probe request from monit - it simply seem to be *missing* a request once in a while)

[Murvin Lai]

Do you run any fs.stat()? there is a report from Node.js on Facebook today sayng there is a problem.  And v0.8.11 will be released tomorrow. 

remote services.. you have to make sure you have large open file (ulimit) available.  However, based on your description about only 2-3 req/sec, that shouldn't be the case.  By, you can watch netstat for # of sockets in use.  

missing request? hmmm do you log all request?  do you have process.on('uncaughtException', function(){}) to log anything thrown without going to the res?

[Hugues Hardel]

* The server doesn't use fs.stat() under normal processing (aka certain admin command will cause the server to use fs.stat but the command are rarely used if ever)

* We had set ulimit to unlimited as a precaution, I don't think this would be an issue but I will keep an eye on netstat the next time we try out v0.8.x

* Yes, we log all incoming requests before any other processing, we listen & log to the server's error event & and we log everything catched by the 'uncaughtException' event.

With the server not acknowledging the request there is isn't much to go by...I think I'll try to reproduce monit's probe request issue with a homegrown client, and investigate the problem from the client's perspective

[Murvin Lai]

hmmm i can't think of anything else ..that's kinda hard to debug. 

please update me if  you find anything.  I wanna know too. :)  

[Ben Noordhuis]

Have you looked at the API changes wiki page[1]?

[1] https://github.com/joyent/node/wiki/API-changes-between-v0.6-and-v0.8

[Zane Claes]

FWIW, I have an open StackOverflow question which might be related to this issue, where I've done a lot of debugging ...

http://stackoverflow.com/questions/12886438/node-js-app-has-periodic-slowness-and-or-timeouts-does-not-accept-incoming-requ

[Hugo]

Hi Zane

My problems go away immediately after I swap out node to v8.x for v6.18. If you have the luxury to swap out node.js versions in prod it would be interesting for you to try 6.18...if you server runs ok then your application code isn't the problem, and we likely are experiencing the same issue