Memory leak, growing RSS and debugging

[Nico Kaiser]

Hi!

For some months I'm desperately searching for a memory leak in my Node server.

It's a simple publish/subscribe WebSocket server that serves about 15-20.000 concurrent clients. 

With Node 0.4 and miksago's node-websocket-server library the memory consumption was ok, and the memory was freed again at night, when there were only a few clients (about 2.000).

However after upgrading to Node 0.6  OR  using a different library (WebSocket.IO, ws, websock or faye-websocket - Anything that supports the new hybi protocol), the server starts to leak memory.

At startup it consumes needs about 20 MB memory (RSS), with 15.000 clients memory grows to about 250 MB. That's ok so far, however the RSS grows at about 100 MB per hour, which is far too much.

How can I debug this behavior? I tried node-inspector, but v8-profiler only works with Node 0.4. Running node-gc every few seconds smoothes the memory curve, but does not help anything.

I observed that process.memoryUsage().heapTotal stays at an acceptable level (about 150 MB), only rss grows until the process gets killed. Does this tell anything, e.g. Buffer leak? (I know the WebSocket modules use lots of small buffers, but they should get freed, shouldn't they?).

Do you have any hint where I can look for leaks?

Thanks,

Nico

[Ilya Dmitrichenko]

On 19 March 2012 14:09, Nico Kaiser <ni...@kaiser.me> wrote:
> How can I debug this behavior? I tried node-inspector, but v8-profiler only
> works with Node 0.4. Running node-gc every few seconds smoothes the memory
> curve, but does not help anything.

If you did read the list, you would come across the same issue being
discussed in this post:
https://groups.google.com/d/msg/nodejs/Aq9BId5Tff8/7G3JxaIVd2YJ

https://github.com/Jimbly/node-mtrace

> I observed that process.memoryUsage().heapTotal stays at an acceptable level
> (about 150 MB), only rss grows until the process gets killed. Does this tell
> anything, e.g. Buffer leak? (I know the WebSocket modules use lots of small
> buffers, but they should get freed, shouldn't they?).
>
>
> Do you have any hint where I can look for leaks?
>
> Thanks,
> Nico
>

> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en

[Nico Kaiser]

Thanks Ilya, I'll have a second look at node-mtrace. 

I tried this (as I did read the list ;-)), but it did not help much, and I hoped someone could identify a specific class of problems with my description (RSS grows, heap stays ok)...

Nico

> nodejs+unsubscribe@googlegroups.com

[Jimb Esser]

Rising RSS and no JS heap growth is exactly the symptoms I was seeing
that caused me to expose mtrace in a node module, but my conclusion is
that it's a hard thing to track down, especially in C++ app. Sadly it
seems Buffers show up as just "operator new()" in an mtrace, so if
those are leaking, you won't know much for sure, as it'll just be
reported in the one big "new" bucket, though looking at the sizes of
outstanding allocations (using the GCC mtrace command line tool) you
might be able to deduce something useful.

I also had some luck just taking a core dump with gcore and looking at
the dump in a hex editor (in my case, the data was clearly big blocks
of triplets of 32 bit floats and then big blocks of integers, so I
could conclude it was vertex and index buffers, and combined with the
mtrace results I knew to look for places that were calling new on
these kinds of objects and might not be deleting them - in your case
perhaps you'll get lucky and see some repeating data that might
indicate what kind of data is leaking).

It's useful to note that any Buffers under 8kb in size end up coming
out of pooled 8kb buffers, and as long as any JS object is referencing
any slice of that buffer, the whole 8kb is still going to be in
memory, which can be misleading (or the cause of "leaks" if you're,
say, slicing out a few bytes of the buffer and hanging on to them for
some reason).

- Jimb

> > > nodejs+un...@googlegroups.com

[wavded]

Nico did you have any luck in your debugging adventures?  My symptoms seems to very closely match yours.  Was going to try mtrace.

> nodejs+un...@googlegroups.com

[Stefan Zehe]

Same problem here. Our app slowly increases the rss. and i could not find anything with node-webkit-agent :(

Mit freundlichen Grüßen

-- 
Stefan Zehe
0951 - 510 908 106
------------------------------
upjers GmbH & Co. KG
Hafenstr. 13
96052 Bamberg

Sitz der Gesellschaft: Bamberg
Registergericht: Bamberg, HRA 10694
Gesellschafter: Upjers Verwaltungs GmbH

http://www.upjers.com
------------------------------

[Stefan Zehe]

i solved my problems with an update to 0.8.9.

in 0.8.3 there was a bugfix

> events: Fix memory leak from removeAllListeners (Nathan Rajlich)

which might caused a steady growing _events-array.

[Nico Kaiser]

Nothing new, even with node 0.8.8.

However it does not feel like a "leak" (where RSS grows until the process crashes), it just fills "most" of the system memory and then stays at this level (even when all the connections are closed). I suspect V8 memory management to be the culprit.

Nico

[Nico Kaiser]

Am 13.09.2012 um 10:05 schrieb Stefan Zehe <s.z...@upjers.com>:

> i solved my problems with an update to 0.8.9.
>
> in 0.8.3 there was a bugfix
>
> > events: Fix memory leak from removeAllListeners (Nathan Rajlich)
>
> which might caused a steady growing _events-array.

I'll check if I can get information about the _events arrays, but as I wrote before, 0.8.8 did not change much:

http://cl.ly/image/2Y3N3w0t1I34


Nico

[Tristan Slominski]

Having similar issue to what Nico is seeing... 

(a not very helpful picture follows):

the blue is RSS growing to fill up available memory, while the tiny pink and red lines are the heapTotal and heapUsed, which stays bounded (around 20-25 MB heap total in my case), data from 0.8.8 (haven't tried 0.8.9 yet)

https://plus.google.com/u/0/photos/106160293361349494652/albums/5787976828393231153/5787976827897869394

the common thread for me with Nico is that connections are constantly created and destroyed. I've seen this behavior both using https connections and then using only tls connections.

[Ben Noordhuis]

On Sun, Sep 16, 2012 at 9:55 PM, Brian Gruber <webb...@gmail.com> wrote:
> I get a similar issue using socket.io with WEBSOCKET transport enabled. It
> also appear that RSS grows to fill up available memory but doesn't seem to
> hit an OOM. It just stays at a very high memory usage. On a 2gig machine it
> climbs to like 1960mb and is quite scary.
>
> I've tried node up to 0.8.9 and still happens.

That's intentional. The V8 garbage collector tries very hard to do as
few sweeps as possible and it does that by growing the heap where
possible. The larger the heap is, the less V8 has to scavenge for
memory when new objects are created.

[Nico Kaiser]

Am 17.09.2012 um 06:42 schrieb Ben Noordhuis <in...@bnoordhuis.nl>:

> On Sun, Sep 16, 2012 at 9:55 PM, Brian Gruber <webb...@gmail.com> wrote:
>> It also appear that RSS grows to fill up available memory but doesn't seem to
>> hit an OOM. It just stays at a very high memory usage.

>> ...

>
> That's intentional. The V8 garbage collector tries very hard to do as
> few sweeps as possible and it does that by growing the heap where
> possible. The larger the heap is, the less V8 has to scavenge for
> memory when new objects are created.

So this is what I was hoping. I agree with Ben, when a machine has enough memory, it should be used.
Lots of free memory are useless ...

So we can "ignore" this "problem", as it is no leak (and thus not causing the process to crash with OOM)...

Nico

[Ryan Schmidt]

On Sep 17, 2012, at 03:09, Nico Kaiser wrote:
> Ben Noordhuis wrote:

>> On Sun, Sep 16, 2012 at 9:55 PM, Brian Gruber wrote:
>>> It also appear that RSS grows to fill up available memory but doesn't seem to
>>> hit an OOM. It just stays at a very high memory usage.
>>> ...
>>
>> That's intentional. The V8 garbage collector tries very hard to do as
>> few sweeps as possible and it does that by growing the heap where
>> possible. The larger the heap is, the less V8 has to scavenge for
>> memory when new objects are created.
>
> So this is what I was hoping. I agree with Ben, when a machine has enough memory, it should be used.
> Lots of free memory are useless ...
>
> So we can "ignore" this "problem", as it is no leak (and thus not causing the process to crash with OOM)...

So do I understand correctly: a node program could use up all the computer's memory?

If I then need to launch additional programs, will node notice this and reduce its memory usage so that the new program doesn't incur virtual memory penalties?

Is there a memory size at which this becomes a performance problem—for example what if I run a node app on a server with 128GB of RAM?

[Ben Noordhuis]

On Mon, Sep 17, 2012 at 3:40 PM, Ryan Schmidt
<googl...@ryandesign.com> wrote:
> So do I understand correctly: a node program could use up all the computer's memory?
>
> If I then need to launch additional programs, will node notice this and reduce its memory usage so that the new program doesn't incur virtual memory penalties?
>
> Is there a memory size at which this becomes a performance problem—for example what if I run a node app on a server with 128GB of RAM?

If you manage to fill up 128 GB, you get a cookie. The 2 GB that was
reported earlier in this thread is close to the maximum that V8 will
allocate.

If that's still too much, you can limit the heap size with
--max-old-space-size=<x> where <x> is in megabytes.

[Ryan Schmidt]

On Sep 17, 2012, at 09:03, Ben Noordhuis wrote:

I like cookies. But let's change the example. Suppose I have a server with 2GB free RAM and I launch a node app. Over time it will occupy all 2GB of free RAM as a consequence of the garbage collection system, even though it's not really using the memory. Now I need to launch another program that needs 1GB of RAM. What will happen?

[Jorge]

Nothing. No problem. You even can have 10 programs with 1 terabyte mapped to each and all will be good... until the programs start to *really* use that memory, then, the OS *may* need to start swapping pages to/from disk like crazy, or not, it depends: if the combined *real* usage is > phisical RAM it will swap, if it isn't, it won't.
--
Jorge.

[wavded]

In case anyone is interested, my memory issue came down to: https://github.com/LearnBoost/socket.io/issues/438 (i ended up setting a websocket only instance (no fallbacks except htmlfile) and xhr only instance, xhr RSS still grows but slower, and websocket will go down when clients disconnect.  all this was happening outside of v8 heap space (seemed like a Buffer leak when using mtrace).

[Vyacheslav Egorov]

V8 also tries to release a completely emoty memory pages back to the OS.

Pages that have some little garbage on them or have a lot of small
holes are subject to compaction.

So in theory amount of unused memory should not exceed roughly 20% in
stable condition... Heap breathes: becomes larger when you actively
allocate and smaller when you are idle. Memory management is full of
heuristics which require careful tweaking... We have tweaked them a
lot since new GC has been released to the wild to make sure that
memory fragmentation stays on acceptable levels.

--
Vyacheslav Egorov

[Brian Gruber]

How did you actually fix the issue, or did you not?