How to handle out-of-memory type of errors.

[Jane Chen]

I'm embedding v8 3.24 in my system.  When running this query:

a = []; for (;;) { a.push("hello"); } 

v8 logs out of memory error and aborts:

#4 0x00007f785148b731 in v8::internal::OS::DebugBreak() () from lib/libv8.so
#5 0x00007f785148bf3d in v8::internal::OS::Abort() () from lib/libv8.so
#6 0x00007f7851109137 in v8::Utils::ReportApiFailure(char const*, char const*) () from lib/libv8.so
#7 0x00007f785110bf5d in v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) () from lib/libv8.so
#8 0x00007f785118f46e in v8::internal::Invoke(bool, v8::internal::Handle<v8::internal::JSFunction>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*, bool*) () from lib/libv8.so
#9 0x00007f785118fd3f in v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*, bool*, bool) () from lib/libv8.so
#10 0x00007f7851115a7e in v8::Script::Run() () from lib/libv8.so 

Ideally I would like to catch this and cancel the request causing the out-of-memory error before v8 aborts.  What's the best way to handle it?

Thanks in advance.

[Ben Noordhuis]

I don't think out of memory is a failure mode that V8 really supports;
it will report an error in most cases but it won't be in a usable
state afterwards.

Maybe you can install a callback with
Isolate::AddMemoryAllocationCallback(), keep a tally of allocated
memory and kill the script before it exhausts all memory. I don't
know if it's safe to call Isolate::TerminateExecution() from a memory
callback but I imagine it is, it just sets a flag.

[Jane Chen]

Ben,

Thanks.  That actually worked for my test case.  Except that my callback is only called once with an apparent size of 1GB.  So no need to tally.  That's my only chance to call TerminateExecution before it aborts.

At least that's what I saw on Linux with v3.24.  Haven't tested it extensively though.  If anyone knows of other scenarios, please share.

Jane

[Jane Chen]

So I found out that after the first malloc callback, it depends on how much v8 can recollect through GC.  In the test case shown above, gc fails to free up anything and the 1GB of memory is all used.  Then it aborts.

In other healthier cases, even if multiple GBs of memory is allocated, as long as GC is able to free up memory, it doesn't abort.

Although I can register GC callbacks, the callback doesn't tell me how successful it is in freeing memory.  --trace_gc does, but not through a callback.  So at this point, I don't have a fix yet.

[Jane Chen]

Here's the trace_gc log:

[13997]      258 ms: Scavenge 2.3 (39.2) -> 1.9 (39.2) MB, 8.4 ms [Runtime::PerformGC].
[13997]      277 ms: Scavenge 2.7 (39.2) -> 2.5 (40.2) MB, 10.3 ms [Runtime::PerformGC].
[13997]      786 ms: Mark-sweep 15.5 (52.2) -> 9.6 (48.5) MB, 186.1 ms (+ 96.5 ms in 1 steps since start of marking, biggest step 96.5 ms) [StackGuard GC request] [GC in old space requested].
[13997]     2189 ms: Mark-sweep 45.1 (84.1) -> 29.3 (68.2) MB, 427.1 ms (+ 276.9 ms in 1 steps since start of marking, biggest step 276.9 ms) [StackGuard GC request] [GC in old space requested].
[13997]     6662 ms: Mark-sweep 149.1 (188.1) -> 95.8 (134.8) MB, 1441.6 ms (+ 903.0 ms in 1 steps since start of marking, biggest step 903.0 ms) [StackGuard GC request] [GC in old space requested].
[13997]    21462 ms: Mark-sweep 500.2 (539.3) -> 320.5 (359.5) MB, 4683.9 ms (+ 2996.5 ms in 1 steps since start of marking, biggest step 2996.5 ms) [StackGuard GC request] [GC in old space requested].
[13997]    35887 ms: Scavenge 1038.9 (1077.9) -> 1038.9 (1077.9) MB, 1299.5 ms (+ 407.7 ms in 1 steps since last GC) [Runtime::PerformGC] [incremental marking delaying mark-sweep].
[13997]    35913 ms: Scavenge 1038.9 (1077.9) -> 1038.9 (1077.9) MB, 1.9 ms (+ 23.4 ms in 1 steps since last GC) [Runtime::PerformGC] [incremental marking delaying mark-sweep].

#
# Fatal error in JS
# Allocation failed - process out of memory
#

[Jane Chen]

Just to close the loop, I found out that for each isolate that's created,

isolate->Enter();

v8::V8::IgnoreOutOfMemoryException();

isolate->Exit();

avoids the crash for the test case I have above.  Not sure if it's safe to always do so.  Still need more testing for other stress situations.

If anyone has more insights or experience dealing with it, I'd appreciate it if you can share them here.

[Ben Noordhuis]

On Sun, Jan 18, 2015 at 5:59 AM, Jane Chen <jxch...@gmail.com> wrote:
> Just to close the loop, I found out that for each isolate that's created,
>
> isolate->Enter();
> v8::V8::IgnoreOutOfMemoryException();
> isolate->Exit();
>
> avoids the crash for the test case I have above. Not sure if it's safe to
> always do so. Still need more testing for other stress situations.
>
> If anyone has more insights or experience dealing with it, I'd appreciate it
> if you can share them here.

Jane, you may want to reconsider using
V8::IgnoreOutOfMemoryException(), it was removed in V8 3.26 (IIRC)
because of [0]. V8 now returns an empty handle on allocation failure
or aborts when that's not possible.

https://code.google.com/p/v8/issues/detail?id=3060

[Jakob Kummerow]

On Sun, Jan 18, 2015 at 5:59 AM, Jane Chen <jxch...@gmail.com> wrote:

Just to close the loop, I found out that for each isolate that's created,

isolate->Enter();

v8::V8::IgnoreOutOfMemoryException();

isolate->Exit();

avoids the crash for the test case I have above.  Not sure if it's safe to always do so.  Still need more testing for other stress situations.

It's not safe. The only safe thing to do in case of OOM is to crash the process. The intention of v8::V8::IgnoreOutOfMemoryException() was to let embedders handle the crashing on their end, but it turned out to be basically impossible to handle all cases in the way they were supposed to be handled, hence...

 

If anyone has more insights or experience dealing with it, I'd appreciate it if you can share them here.

...it has been removed back in March of last year: https://code.google.com/p/v8/source/detail?r=20184, and it's not coming back.

[Jane Chen]

Unfortunately, the following link is no longer valid:

https://code.google.com/p/v8/source/detail?r=20184

We just moved to 4.6.88.  No more IgnoreOutOfMemoryException(), and I am looking for a way to proactively detect low memory condition to terminate a request before it completely runs out of memory.  I looked at 1) GC prologue callbacks; 2) heap statistics; 3) GC traces and 4) the source code.  It seems that heap statistics isn't quite sensitive to the low memory condition.  "last resort gc" seems a more promising indicator, but it may be too late once you get "last resort gc".  However using kGCCallbackFlagForced seems too eager.  Any suggestions on a set of heuristics?

Thanks a lot in advance.

[Jochen Eisinger]

this is the git commit: https://chromium.googlesource.com/v8/v8/+/8b8fb30e7f8915d0762402f0b6e5b16c820cab48

Have you considered setting a FatalErrorCallback and then tearing down the entire isolate and the thread it lived on? That might be better than crashing, although I'm not sure whether you won't run into other problems...

--
--
v8-users mailing list
v8-u...@googlegroups.com
http://groups.google.com/group/v8-users
---
You received this message because you are subscribed to the Google Groups "v8-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to v8-users+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jane Chen]

I already have registered by FatalErrorCallback, but it never gets called from an out of memory condition. 

It would be interesting to get a test case to trigger FatalErrorCallback, since my present handling of it is to gracefully shutdown and die.

[Jochen Eisinger]

With this CL applied: https://codereview.chromium.org/1614763002

I get the following output:

$ ulimit -v 819200

$ out/x64.release/hello-world 

<--- Last few GCs --->

    3308 ms: Scavenge 188.2 (211.0) -> 188.2 (211.0) MB, 84.8 / 0 ms [allocation failure].

    3383 ms: Scavenge 188.2 (211.0) -> 188.2 (211.0) MB, 75.2 / 0 ms [allocation failure].

    3457 ms: Scavenge 188.2 (211.0) -> 188.2 (211.0) MB, 74.2 / 0 ms [allocation failure].

    3514 ms: Scavenge 188.2 (211.0) -> 188.2 (211.0) MB, 56.8 / 0 ms [allocation failure].

    3571 ms: Scavenge 188.2 (211.0) -> 188.2 (211.0) MB, 56.5 / 0 ms [allocation failure].

<--- JS stacktrace --->

Cannot get stack trace in GC.

fatal error: MarkCompactCollector: semi-space copy, fallback in old gen

: Allocation failed - process out of memory

#

# Fatal error in , line 0

# API fatal error handler returned after process out of memory

#

==== C stack trace ===============================

 1: 0x996cb3

 2: 0x407b7b

 3: 0x4078e8

 4: 0x5fd7e0

 5: 0x5f850f

 6: 0x5f8398

 7: 0x5f23e0

 8: 0x5f1ab1

 9: 0x5d5d60

10: 0x5d4d0c

11: 0x5d4554

12: 0x5d42e8

13: 0x5a98a5

14: 0x685bea

15: 0x6836df

16: 0x6a5f82

17: 0x59017f

18: 0x91a56a

19: 0x919c7d

20: 0x5bf3e806458

Illegal instruction (core dumped)

[Jane Chen]

Yes, I do get fatal errors like these from v8, but my fatal error callback is never invoked.  I register my callback after I create the isolate like this:

  isolate->Enter();
  isolate->SetFatalErrorHandler(logFatalError);
  isolate->SetCaptureStackTraceForUncaughtExceptions(true,64,
    v8::StackTrace::kDetailed);
  isolate->AddGCEpilogueCallback(afterGC);
  isolate->Exit();

[Jochen Eisinger]

can you post a standalone repro case? Otherwise, setting a break point in v8's internal fatal error handler and checking why it doesn't invoke yours would be a possible way to debug this.

[Jane Chen]

Jochen,

I do mange to get the callback now, but other than gracefully shutting down, I'm not sure how to tear down the isolate and the thread it lived on, without affecting other isolates and threads.  Is there any sample code or unit test code to demonstrate that?

Thanks a lot in advance.

Jane

[Jochen Eisinger]

in theory, doing isolate->Exit(); isolate->Dispose(); should be enough to discard the isolate. How to terminate the thread depends on the threading library you use. Note that you'll also have to free up all C++ memory you've allocated, because exiting the Isolate won't invoke weak callbacks etc..

[Jane Chen]

Jochen,

That turns out to be really fun exercise, and the result is quite promising.

Thanks!