node server getting stuck

[Lars Jacob]

We recently encountered an issue with our node.js application (based in express.js and socket.io). After a while the application won't respond to any I/O or if only erratically. It still accepts tcp connections (or http connections) but doesn't respond at all. Furthermore it can't establish network connections to the outside world. For example we see "failed to connect to mongodb" which is triggered by a database query which is called within a setInterval callback. And we are seeing this logs only because after a while the server stops to be "locked" and continues to work fine...

Has anybody seen a similar behaviour? I know it's a pretty vague description but until now we were unable to track this down further. Any Ideas?

We could reproduce this with our application in node 0.8.17 and 0.8.22

Thanks for your help in advance!

Lars

[Ryan Schmidt]

On Mar 21, 2013, at 16:26, Lars Jacob wrote:

> We could reproduce this with our application in node 0.8.17 and 0.8.22

Please test with 0.10.1.

[Isaac Schlueter]

> and "console.error" is more of a friend than console.log in locking cases. It is output immediately (so it will slow your server down a tad since it is sync rather than async) but you don't loose any messages

console.log is synchronous as well, has been since 0.6, iirc.



On Thu, Mar 21, 2013 at 2:58 PM, Nathanael Anderson
<nathanae...@gmail.com> wrote:

> Adding lot of logging -- and "console.error" is more of a friend than
> console.log in locking cases. It is output immediately (so it will slow
> your server down a tad since it is sync rather than async) but you don't
> loose any messages. One thing to track is how many connections you have
> (both http & socket) that are currently active and how many connections you
> have active to mongo (any any other things like memcache)... You might be
> finding that one of your pools is not releasing resources.
>
> Nathanael A.
>
> --
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups
> "nodejs" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to nodejs+un...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Lars Jacob]

I already considered high number of open connections, but it's an test environment and "ss -s" and "losf -i" both show < 100 open sockets at any given time.

I also was thinking about the fact that console logs are blocking... We are using winston for logging. I'll try to disable the console transport and let's see how this works out.

[George Snelling]

Isaac, 

That is interesting information.  It is not in the docs afaict: http://nodejs.org/api/stdio.html.  A note in the docs about what happens to in-flight messages to stdio when node crashes would help.  If it's there already and I missed it please accept my headsmack.  

Thanks, 

-g

[Ben Noordhuis]

On Fri, Mar 22, 2013 at 1:19 AM, Isaac Schlueter <i...@izs.me> wrote:
> console.log is synchronous as well, has been since 0.6, iirc.

But only if stdout is a tty or a file, pipes are non-blocking. I've
added a note to the stdio docs in 132c77e.

[Eldar]

I recently benchmarked node.js hello world server on OS X 10.8.3 and observed the following:

$ node hello-world.js &

$ ab -n 1000 -c 10 -k -q http://127.0.0.1:8000/ # OK

$ ab -n 2000 -c 10 -k -q http://127.0.0.1:8000/ # OK

$ ab -n 5000 -c 10 -k -q http://127.0.0.1:8000/ # Hang out!

That's probably not related to the above issue, since it's only OS X specific (other platforms are ok), but still interesting.

The behavior is the same for both 0.8.x and 0.10.x. 

[Ben Noordhuis]

On Sat, Mar 23, 2013 at 8:13 PM, Eldar <elda...@gmail.com> wrote:
> I recently benchmarked node.js hello world server on OS X 10.8.3 and
> observed the following:
>
> $ node hello-world.js &
> $ ab -n 1000 -c 10 -k -q http://127.0.0.1:8000/ # OK
> $ ab -n 2000 -c 10 -k -q http://127.0.0.1:8000/ # OK
> $ ab -n 5000 -c 10 -k -q http://127.0.0.1:8000/ # Hang out!
>
> That's probably not related to the above issue, since it's only OS X
> specific (other platforms are ok), but still interesting.
> The behavior is the same for both 0.8.x and 0.10.x.

Try a tool like siege or wrk, the ab that Apple ships is buggy.

[Eldar]

Yeah, with wrk everything is fine, thanks!