Running allocation failed - out of memory at 8MB

[Jos Kuijpers]

Hi All,

Yesterday I implemented a JS module loader in my system when I ran into the out of memory message:

#
# Fatal error in JS
# Allocation failed - process out of memory
#

This is when the process uses only 8MB! And when I add some native code causing me to run at 15MB, it crashes at the same location...

So I wonder, what else (other than actually no memory) could cause this problem? Some heap corruption? How could I have caused this?

The code can be found at https://github.com/joskuijpers/L8Framework and https://github.com/joskuijpers/Andromeda. I really have no clue in what direction I should look so I can create a minimal testcase for you.

Thanks in advance,

Jos

[Ben Noordhuis]

Just a hunch but V8 uses mmap(2) to allocate virtual memory in large
chunks. If your address space is fragmented, then there may not be a
contiguous range large enough to fulfill the request. It's really
only an issue with 32 bits architectures because the x64 address space
is so much larger.

You can check for failing mmap system calls with dtruss or dtrace
(assuming you're on OS X, which I suspect you are.)

[Jos Kuijpers]

I like the hunch, but no, it seems not:

$ sudo dtruss -n Andromeda -t mmap

shows many calls, but no mmap call returning 0x0. It is hard to see when exactly the fatal error comes and with what MMAP that happened.

[Ben Noordhuis]

On Sat, Apr 19, 2014 at 4:16 PM, Jos Kuijpers <j...@kuijpersvof.nl> wrote:
> I like the hunch, but no, it seems not:
>
> $ sudo dtruss -n Andromeda -t mmap
>
> shows many calls, but no mmap call returning 0x0. It is hard to see when
> exactly the fatal error comes and with what MMAP that happened.

mmap returns -1 / 0xffffffff on error, not 0.

[Jos Kuijpers]

All the same:

84258/0x45cd5d:       472      27     13 mmap(0x100058000, 0x14000, 0x5, 0x12, 0x3, 0x0) = 0x100058000 0

84258/0x45cd5d:       479      15      5 mmap(0x10006C000, 0x6000, 0x3, 0x12, 0x3, 0x14000) = 0x10006C000 0

84258/0x45cd5d:       483      13      3 mmap(0x100072000, 0xCB80, 0x1, 0x12, 0x3, 0x1A000) = 0x100072000 0

84258/0x45cd5d:       686      22     10 mmap(0x10007F000, 0x10E3000, 0x5, 0x12, 0x3, 0x0) = 0x10007F000 0

84258/0x45cd5d:       692      14      5 mmap(0x101162000, 0x45000, 0x3, 0x12, 0x3, 0x10E3000) = 0x101162000 0

84258/0x45cd5d:       698      13      4 mmap(0x1011AD000, 0x4773CC, 0x1, 0x12, 0x3, 0x1128000) = 0x1011AD000 0

84258/0x45cd5d:   1671677      23      8 mmap(0x0, 0x1000, 0x1, 0x1, 0x4, 0x0) = 0x1016AF000 0

84258/0x45cd5d:   1676178      27     10 mmap(0x0, 0x16726F0, 0x1, 0x1, 0x4, 0x0) = 0x103000000 0

84258/0x45cd5d:   1711220      33     20 mmap(0x123441B36000, 0x6E4C0, 0x1, 0x12, 0x4, 0x1B22000) = 0x123441B36000 0

84258/0x45cd5d:   1711520   18804     20 mmap(0x106000000, 0x74B000, 0x5, 0x12, 0x4, 0x1000) = 0x106000000 0

84258/0x45cd5d:   1711538      29     13 mmap(0x10674B000, 0x60000, 0x3, 0x12, 0x4, 0x74C000) = 0x10674B000 0

84258/0x45cd5d:   1711545      15      5 mmap(0x1067D0000, 0x1DC00, 0x1, 0x12, 0x4, 0x7AC000) = 0x1067D0000 0

84258/0x45cd5d:   1711743      16      6 mmap(0x10228F000, 0xC000, 0x5, 0x12, 0x4, 0x1000) = 0x10228F000 0

84258/0x45cd5d:   1711749      15      5 mmap(0x10229B000, 0x1000, 0x3, 0x12, 0x4, 0xD000) = 0x10229B000 0

84258/0x45cd5d:   1711754      14      4 mmap(0x10229C000, 0x4D60, 0x1, 0x12, 0x4, 0xE000) = 0x10229C000 0

84258/0x45cd5d:   1874886      20      5 mmap(0x0, 0x17000, 0x3, 0x1002, 0x34000000, 0x0) = 0x102563000 0

84258/0x45cd5d:   1875236      13      3 mmap(0x0, 0x17000, 0x3, 0x1002, 0x36000000, 0x0) = 0x10257A000 0

84258/0x45cd5d:   1875288      12      2 mmap(0x0, 0x3000, 0x3, 0x1002, 0x34000000, 0x0) = 0x1022FA000 0

84258/0x45cd5d:   1875525      13      3 mmap(0x0, 0x3000, 0x3, 0x1002, 0x36000000, 0x0) = 0x1022FD000 0

84258/0x45cd5d:   1672246      21      8 mmap(0x0, 0x1000, 0x1, 0x1, 0x4, 0x0) = 0x1016B0000 0

84258/0x45cd5d:   1672350      15      5 mmap(0x0, 0x1000, 0x1, 0x1, 0x4, 0x0) = 0x1016B1000 0

84258/0x45cd5d:   1681606      19      7 mmap(0x0, 0x1000, 0x1, 0x1, 0x4, 0x0) = 0x102682000 0

84258/0x45cd5d:   1690140      23     10 mmap(0x0, 0x1DE3, 0x1, 0x2, 0x4, 0x0) = 0x102289000 0

84258/0x45cd5d:   1693918      24     11 mmap(0x0, 0x8F2FA8, 0x1, 0x2, 0x4, 0x0) = 0x104AEF000 0

84258/0x45cd5d:   1694451      51     21 mmap(0x0, 0xC48, 0x1, 0x2, 0x4, 0x0) = 0x10228B000 0

84258/0x45cd5d:   1706761      20      4 mmap(0x0, 0x1000, 0x3, 0x1002, 0xFFFFFFFF, 0x0) = 0x10228D000 0

84258/0x45cd5d:   1706772      13      2 mmap(0x0, 0x1000, 0x3, 0x1002, 0xFFFFFFFF, 0x0) = 0x10228E000 0

84258/0x45cd5d:   1711173   24399     24 mmap(0x123440000000, 0x86E000, 0x5, 0x12, 0x4, 0x1000) = 0x123440000000 0

84258/0x45cd5d:   1711194      32     15 mmap(0x12344086E000, 0x12B3000, 0x3, 0x12, 0x4, 0x86F000) = 0x12344086E000 0

84258/0x45cd5d:   1876993      30     15 mmap(0x0, 0x52C982, 0x3, 0x2, 0x4, 0x0) = 0x1067EE000 0

84258/0x45cd5d:   1877197      15      5 mmap(0x0, 0x7A8, 0x1, 0x2, 0x6, 0x0) = 0x1022FA000 0

84258/0x45cd5d:   1877353      13      3 mmap(0x0, 0x1000, 0x3, 0x1002, 0x34000000, 0x0) = 0x1022FB000 0

84258/0x45cd5d:   1877539      14      5 mmap(0x0, 0x7BC, 0x1, 0x2, 0x6, 0x0) = 0x1022FC000 0

84258/0x45cd5d:   1878602      56     24 mmap(0x0, 0x111C, 0x1, 0x2, 0x6, 0x0) = 0x102564000 0

84258/0x45cd5d:   1881638      27     12 mmap(0x0, 0x30BAA8, 0x1, 0x402, 0x6, 0x0) = 0x1053E2000 0

84258/0x45cd5d:   1884845      18      5 mmap(0x0, 0x1000, 0x3, 0x1002, 0x34000000, 0x0) = 0x102563000 0

84258/0x45cd5d:   1886032      25     13 mmap(0x0, 0x42B660, 0x1, 0x2, 0x6, 0x0) = 0x106D1B000 0

dtrace: 518 dynamic variable drops with non-empty dirty list

84258/0x45cd5d:   1942287      20      6 mmap(0x298CDB1DD000, 0x1000, 0x7, 0x1002, 0xFF000000, 0x0) = 0x298CDB1DD000 0

84258/0x45cd5d:   1942610      13      3 mmap(0x342523071000, 0x4000000, 0x0, 0x1042, 0xFF000000, 0x0) = 0x342523071000 0

84258/0x45cd5d:   1942630      13      4 mmap(0x342524000000, 0x100000, 0x3, 0x1012, 0xFF000000, 0x0) = 0x342524000000 0

84258/0x45cd5d:   1942671      12      3 mmap(0x1C9B35D82000, 0x20000000, 0x0, 0x1042, 0xFF000000, 0x0) = 0x1C9B35D82000 0

84258/0x45cd5d:   1942708      11      2 mmap(0x1E74D4D76000, 0x60000, 0x0, 0x1042, 0xFF000000, 0x0) = 0x1E74D4D76000 0

84258/0x45cd5d:   1942711      10      1 mmap(0x346EF9A12000, 0x200000, 0x0, 0x1042, 0xFF000000, 0x0) = 0x346EF9A12000 0

84258/0x45cd5d:   1942719      12      3 mmap(0x346EF9A12000, 0x1000, 0x3, 0x1012, 0xFF000000, 0x0) = 0x346EF9A12000 0

84258/0x45cd5d:   1942723      12      3 mmap(0x1E74D4D80000, 0x20000, 0x3, 0x1012, 0xFF000000, 0x0) = 0x1E74D4D80000 0

84258/0x45cd5d:   1942854      16      6 mmap(0x1C9B35E00000, 0x5000, 0x3, 0x1012, 0xFF000000, 0x0) = 0x1C9B35E00000 0

84258/0x45cd5d:   1942862      12      3 mmap(0x1C9B35E06000, 0x1000, 0x7, 0x1012, 0xFF000000, 0x0) = 0x1C9B35E06000 0

84258/0x45cd5d:   1942905      13      4 mmap(0x1C9B35F00000, 0x5000, 0x3, 0x1012, 0xFF000000, 0x0) = 0x1C9B35F00000 0

84258/0x45cd5d:   1942911      12      3 mmap(0x1C9B35F06000, 0x1000, 0x7, 0x1012, 0xFF000000, 0x0) = 0x1C9B35F06000 0

84258/0x45cd5d:   1942940      12      3 mmap(0x1C9B36000000, 0x5000, 0x3, 0x1012, 0xFF000000, 0x0) = 0x1C9B36000000 0

84258/0x45cd5d:   1942947      13      3 mmap(0x1C9B36006000, 0x1000, 0x7, 0x1012, 0xFF000000, 0x0) = 0x1C9B36006000 0

84258/0x45cd5d:   1943209      14      3 mmap(0xCF306D44000, 0x195000, 0x0, 0x1042, 0xFF000000, 0x0) = 0xCF306D44000 0

84258/0x45cd5d:   1943218      12      3 mmap(0xCF306E00000, 0x95000, 0x3, 0x1012, 0xFF000000, 0x0) = 0xCF306E00000 0

84258/0x45cd5d:   1943254      12      2 mmap(0x28227646000, 0x135000, 0x0, 0x1042, 0xFF000000, 0x0) = 0x28227646000 0

84258/0x45cd5d:   1943263      12      2 mmap(0x28227700000, 0x35000, 0x3, 0x1012, 0xFF000000, 0x0) = 0x28227700000 0

84258/0x45cd5d:   1943299      14      4 mmap(0x1C9B36100000, 0x5000, 0x3, 0x1012, 0xFF000000, 0x0) = 0x1C9B36100000 0

84258/0x45cd5d:   1943306      13      3 mmap(0x1C9B36106000, 0xF9000, 0x7, 0x1012, 0xFF000000, 0x0) = 0x1C9B36106000 0

84258/0x45cd5d:   1943356      11      2 mmap(0x27615EB5B000, 0x125000, 0x0, 0x1042, 0xFF000000, 0x0) = 0x27615EB5B000 0

84258/0x45cd5d:   1943364      11      2 mmap(0x27615EC00000, 0x25000, 0x3, 0x1012, 0xFF000000, 0x0) = 0x27615EC00000 0

84258/0x45cd5d:   1943396      11      2 mmap(0x303B17CB5000, 0x125000, 0x0, 0x1042, 0xFF000000, 0x0) = 0x303B17CB5000 0

84258/0x45cd5d:   1943408      15      7 mmap(0x303B17D00000, 0x25000, 0x3, 0x1012, 0xFF000000, 0x0) = 0x303B17D00000 0

84258/0x45cd5d:   1946088      31      9 mmap(0x1D77A121F000, 0x115000, 0x0, 0x1042, 0xFF000000, 0x0) = 0x1D77A121F000 0

84258/0x45cd5d:   1946100      12      3 mmap(0x1D77A1300000, 0x15000, 0x3, 0x1012, 0xFF000000, 0x0) = 0x1D77A1300000 0

84258/0x45cd5d:   1928623      38     23 mmap(0x0, 0xD6A60, 0x1, 0x2, 0x8, 0x0) = 0x1089C3000 0

[Jos Kuijpers]

Is there anything else I can try, or that could be going on? Any way for me to provide to you more information?

[Ben Noordhuis]

On Sat, Apr 19, 2014 at 11:58 PM, Jos Kuijpers <j...@kuijpersvof.nl> wrote:
> Is there anything else I can try, or that could be going on? Any way for me
> to provide to you more information?

There are a number of ways to go about it but my usual approach is to
compile a debug build (`make x64.debug`) and put a breakpoint on
V8::FatalProcessOutOfMemory() in api.cc. That function collects
relevant metrics in a HeapStats struct that you can inspect.

When you compile V8 with gdb support (pass gdbjit=on to make), then
`backtrace` will also print function names of JS stack frames.

[Jos Kuijpers]

This might be a stupid question, but how to do the gdbjit=on part for xcodebuild? :)

[Ben Noordhuis]

On Sun, Apr 20, 2014 at 12:02 PM, Jos Kuijpers <j...@kuijpersvof.nl> wrote:
> This might be a stupid question, but how to do the gdbjit=on part for
> xcodebuild? :)

Not sure but you can always hack the appropriate .gypi file and set
v8_enable_gdbjit to 1.

[Jos Kuijpers]

Found it: -Dv8_enable_gdbjit=1 on GYP_DEFINES. I missed it in the manual on google code.
The Wiki page says it only works on Linux btw. is that out-dated?(like everything else?)

Anyways, I recompiled, made it work, set a breakpoint. Somehow I get now a different error:

# Fatal error in ../../src/handles-inl.h, line 68

# CHECK(location_ != __null && !(*location_)->IsFailure()) failed

I am going to replace some JS code and see if it originates from there. But I don't think that any JS code should be able to cause a crash of V8.

Btw, I run 3.24.40.

[Jos Kuijpers]

So I found the problem.

I store a Local<Value> in my L8Value wrapper class, and then stored the L8Value somewhere (strong). In some code, I cached L8Values. The v8 values it held however already expired because no strong holder existed. I now made the L8Value handle to the V8 Value strong using a Persistent handle. And this works.

[Flying Jester]

On Sunday, April 20, 2014 4:37:28 AM UTC-8, Jos Kuijpers wrote:

Found it: -Dv8_enable_gdbjit=1 on GYP_DEFINES. I missed it in the manual on google code.
The Wiki page says it only works on Linux btw. is that out-dated?(like everything else?)

It most definitely is out of date. I am pretty sure it works on Windows, too, and I would be surprised if it didn't work on OS X.