Memory leak from disposed isolates?

[Jane Chen]

Related to this thread:

  https://groups.google.com/forum/#!topic/v8-users/deYE9pzK9b8

I am having some difficulty with garbage collection in v8 5.3.  As a result, I do the following after running a JavaScript request:

1) ContextDisposedNotification
2) IdleNotificationDeadline
3) If uncollected references still exist, LowMemoryNotification
4) If uncollected references still exist, reset them in my code, then dispose the isolate.

Although this avoids a crash or memory corruption most of the time, I get a memory leak and eventually run out of memory.

The way I dispose an isolate is just by calling Isolate::Dispose().  Am I missing anything?

Conceptually, is the memory allocated and consumed in an isolate completely freed when the isolate is disposed?  Even when garbage collection is incomplete in that isolate?

Thanks in advance.

[Ben Noordhuis]

It should release all resources, yes. I was looking into
multi-isolate support in node.js a while ago and I didn't notice any
memory leaks with 5.3 (IIRC, it might also have been 5.1.)

Have you run your application through valgrind? Is it possible it's
memory fragmentation instead of a memory leak?

If you have multiple isolates making interleaved allocations, it's
possible the address space fragments too much for a new isolate to
reserve a sufficiently large chunk, even if some of the older isolates
have been disposed of.

That's normally only a problem with 32 bits builds although 64 bits
Solaris/Illumos can be pretty terrible too.

[Jane Chen]

Thanks Ben.  I'm testing this on Linux.  RH7 VM and Fedora 20.  It's good to know that you haven't seen this yourself. 
I am seeing it when I dispose the isolate only when LowMemoryNotification is insufficient to clear stuff. 

I am sure it's a leak not a fragmentation.  This is what I see with sar -r:

01:00:02 PM 14200464 18580952 56.68 192932 4271524 31294112 76.01 14207024 3105796 156
01:10:01 PM 9128756 23652660 72.15 193988 4310140 37980896 92.25 19236772 3136272 136
01:20:02 PM 5579180 27202236 82.98 194800 4255556 41479544 100.75 22840464 3074896 144
01:30:02 PM 778656 32002760 97.62 175880 4218864 46678872 113.38 27647292 3056648 96
01:40:02 PM 251640 32529776 99.23 2372 581720 54419028 132.18 29623748 1665516 144
01:50:02 PM 405564 32375852 98.76 2516 499560 58361172 141.76 28903920 2223420 148
--- process got killed and restarted ---
02:00:02 PM 29851432 2929984 8.94 13224 696908 15415752 37.44 1416408 386944 0

I'll try with a different test to dispose and recreate isolates that don't have issues with GC.  I'll report back.

[Jane Chen]

For experiment, I just changed my implementation to always dispose and recreate isolates after each JavaScript execution.  I can easily reproduce a memory leak and my server process runs out of memory within an hour.

If I reuse the isolates, the server holds up for days executing the same set of requests.

[Ben Noordhuis]

On Wed, Dec 14, 2016 at 2:47 AM, Jane Chen <jxch...@gmail.com> wrote:
> For experiment, I just changed my implementation to always dispose and
> recreate isolates after each JavaScript execution. I can easily reproduce a
> memory leak and my server process runs out of memory within an hour.
>
> If I reuse the isolates, the server holds up for days executing the same set
> of requests.

Can you check with valgrind?

[Jane Chen]

I did. I didn't find anything related to isolate disposal.

I tried creating and disposing isolates in a loop in v8_shell and cannot reproduce a memory leak. I'm trying other things to narrow down the problem.

[Jane Chen]

I found the memory leak is caused by the following two flags: gdbjit_full and harmony rest parameters.  Removing them fixed the problem.

Thanks Ben for your help.