node.js web-server hangups

[zheg...@enaza.ru]

Hi.

I'm working as system engineer in a company developing various software. One of the project I'm involved in uses a web server implemented with node.js, and I kinda have a problem there. I run it on a Solaris 11 x 86 server, node version is 0.10.28. About 10% of time it doesn't serve incoming requests. I'm running an nginx in front of it, and it looks like nginx is getting timeouts while reading from node (connection is establishing just fine). I don't have any skills in node or javascript other than installing/configuring, so I took the dtrace toolkit and ran a script that shows the systemcalls issued by the node process (dtruss, actually). Looks like when hanging the process issues multiple successful nanosleep() calls,and a number of periodic lwp_park() calls which get an error 62. This physical server also runs multiple network services (for years, actually; and they remain operational during node web server outages) so I don't think this can be related to some global OS misconfiguration on the OS level.
Is this a javascript code issue, a node issue or something else ?

Thanks.
Eugene.

[Aria Stewart]

i’d suspect a bug in what node’s running, most likely. node has better integration with dtrace, so you should be able to probe a little more finely than system calls there.

Hanging could be not calling back with the data for whatever reason — accepting a connection but not responding is exactly what happens when a callback is dropped by the application.

Tough to diagnose more without more information, but that’s where I’d start.

Aria

[Bruno Fuster]

That 10% happens on specific calls or randomly?

If the CPU usage is too high node won't be able to answer but it still accept incoming connections. Also happens if there's a huge leak and memory consumption is too high as well.

Start by monitoring those processes and check CPU usage and memory. You could also take advantage of newrelic to understand how long requests are taking to respond and where the bottleneck is (mongo is taking too long to answer or MySQL or whatever).

I have seen issues where an unindexed query hangs mongo which hangs node, same could happen on redis.

Hope it helps
Cheers