Re: [nodejs] Excessive memory usage ratio

[Marak Squires]

It's possible you've got a leak in your application code ( accidentally not calling .end on a response stream is a common issue I've seen ).

There are profiling tools to help track this sort of thing down, but I'm not sure which is the best these days. Usually, I find that a code review is enough to track down this sort of thing.

It's also possible there is a leak in core, or in one of the third-party modules you are using. Further investigation seems warranted.

On Mon, Jul 30, 2012 at 12:44 PM, Rusty <gic...@gmail.com> wrote:

Howdy folks,

I have a node program using express, socket.io, and amqp.

It seems to grow in memory usage very quick under load.  By the time it reaches about 25,000 events complete, process.memoryUsage() reports the following.

{ rss: 1072037888, heapTotal: 54075392, heapUsed: 30898800 }

That is about 1GB of RSS and 54MB of total heap.  The process is almost 20x the heap size!

Does that scare anyone else? :-/

During the time it is growing, the JS heap usage fluctuates between 10MB and 42MB, depending on the number of clients connected and the type of events being processed.

I am currently running v0.8.4, but I saw the same problem on v0.8.2 and v0.8.3.

Anyone else seeing memory issues like this?

~Rusty

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

[Rusty]

On Monday, July 30, 2012 6:55:23 PM UTC-7, Marak Squires wrote:

It's possible you've got a leak in your application code ( accidentally not calling .end on a response stream is a common issue I've seen ).

Thanks for your help Marak.

The code uses socket.io, and in my testing I have it serving only 8 clients so the number of connections opened should be constant.  I am not opening any connections myself beyond the basic usage.

I will work on a test case to narrow things down.

 

There are profiling tools to help track this sort of thing down, but I'm not sure which is the best these days. Usually, I find that a code review is enough to track down this sort of thing.

Code review is great, and this is all after a thorough review and cleanup.  However, it seems like a lack of tools for this situation is a significant drawback to this platform.

All of the memory tools I am aware of, I have tried.  All the V8 heap tools are of little use since the JS heap isn't growing.  The distribution of types across the heap is relatively constant as well (as far as I can see).

Valgrind is great for "real" leaks in "native" code, but it doesn't really help here.

 

It's also possible there is a leak in core, or in one of the third-party modules you are using. Further investigation seems warranted.

  

Hopefully I can come up with a reduction that explains what is going on. :-/

The more problems I find like this, the more I realize why people enjoy there silly "stable" environments. ;-)

Thanks again Marak.

~Rusty

[Jimb Esser]

You may want to try mtrace, I've exposed it in a node module here:

https://github.com/Jimbly/node-mtrace

It allows you to trace and analyze native heap memory usage, though if most of the native code you're in is C++, it's just going to say all the memory was allocated by "operator new()", but it's worth a shot.