How do I not abort on OOM?

[Kenton Varda]

Hi v8-users,

It appears that in some cases V8 will abort the process when it runs out of heap space rather than throw a JS exception. The behavior can be overridden by registering an OOM callback, but if that callback returns without aborting, it seems V8 promptly crashes.

It seems like some code paths are designed to handle OOM gracefully, but others aren't.

For my use case, it's pretty important that a malicious script cannot cause the process to abort, since our processes are multi-tenant. Ideally OOM would throw an exception, but terminating the isolate is also acceptable, as long as other isolates can keep going.

Is there any way to accomplish this?

For example, what if I compile with C++ exceptions enabled, and have my OOM handler throw an exception, hence unwinding the stack back to where I entered V8. Then, I promptly destroy the isolate. Would that work?

Or, is there some trick to making V8 less crashy on OOM, aside from going through and fixing all the code paths that crash (which probably isn't feasible for me)?

Thanks,

-Kenton

[Ben Noordhuis]

On Tue, Oct 24, 2017 at 10:17 PM, 'Kenton Varda' via v8-users
<v8-u...@googlegroups.com> wrote:
> Hi v8-users,
>
> It appears that in some cases V8 will abort the process when it runs out of
> heap space rather than throw a JS exception. The behavior can be overridden
> by registering an OOM callback, but if that callback returns without
> aborting, it seems V8 promptly crashes.
>
> It seems like some code paths are designed to handle OOM gracefully, but
> others aren't.
>
> For my use case, it's pretty important that a malicious script cannot cause
> the process to abort, since our processes are multi-tenant. Ideally OOM
> would throw an exception, but terminating the isolate is also acceptable, as
> long as other isolates can keep going.
>
> Is there any way to accomplish this?

No. Graceful handling of OOM conditions is not one of V8's design goals.

> For example, what if I compile with C++ exceptions enabled, and have my OOM
> handler throw an exception, hence unwinding the stack back to where I
> entered V8. Then, I promptly destroy the isolate. Would that work?

No. It would end very badly. V8 is not exception-safe.

> Or, is there some trick to making V8 less crashy on OOM, aside from going
> through and fixing all the code paths that crash (which probably isn't
> feasible for me)?

No tricks, no. The best you can do is monitor memory usage and call
`Isolate::TerminateExecution()` when it gets too high but that won't
be 100% reliable; OOM conditions in C++ code will still be fatal.

Probably not the answers you were hoping for but there it is.

[Andreas Rossberg]

To supplement Ben's answer: the reason why this was never a design goal is that it is practically impossible to achieve. It is extremely difficult to recover reliably from OOM in certain situations. AFAIK, no engine out there is able to guarantee that.

--
--
v8-users mailing list
v8-u...@googlegroups.com
http://groups.google.com/group/v8-users
---
You received this message because you are subscribed to the Google Groups "v8-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to v8-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ken...@cloudflare.com]

Hi all,

Sorry, I think my first message was confusing.

To be clear, in my case, my C++ process is nowhere near being out of memory. But, I'm configuring the V8 isolates I create with a low heap size, using v8::ResourceConstraints::set_max_old_space_size(). It's the V8 heap that is running out of memory.

I'm well aware that writing C++ code that can recover from malloc() failure is basically impossible.

But in my case, V8 is hitting a self-imposed limit on specifically the V8 heap. It seems like we should be able to find a hack to work around this.

For example, I could live with a solution like: When V8's heap is exhausted, it calls the OOM callback. If that returns without terminating, then V8 doubles its own heap limit and continues.

In this case, I would have my OOM callback call TerminateExecution() and arrange for the isolate to be discarded as soon as control returns back out of V8. It's fine if extra memory is allocated in the meantime as a stopgap to avoid aborting.

Is there a way to accomplish something like this? If not, how hard would it be to add?

>> For example, what if I compile with C++ exceptions enabled, and have my OOM 

>> handler throw an exception, hence unwinding the stack back to where I 

>> entered V8. Then, I promptly destroy the isolate. Would that work? 

>

> No.  It would end very badly.  V8 is not exception-safe. 

To be clear about this suggestion: Obviously it's not going to be clean, but I was hoping to discuss what kind of bad things would actually be likely to happen. The isolate would be left in an inconsistent state, but if I'm deleting it anyway, how much does that matter? Small memory leaks would not be a big deal here, since I can periodically drain the process and start a new one to "fix" them.

-Kenton

To unsubscribe from this group and stop receiving emails from it, send an email to v8-users+u...@googlegroups.com.

[Prakash Bailkeri]

Hello Kenton,

Did you find a reliable way to resolve "isolate heap" OOM issue? I had similar question and proposing to use AddNearHeapLimitCallback?

https://groups.google.com/forum/?nomobile=true#!topic/v8-users/qLs7-XT2Zvg

I am trying to solve a issue where hacker/"accidental code"/"bug" asks for memory more than the heap memory limit on isolate (set_max_old_space_size)

Any feedback from you?

Thanks, Prakash 

[Kenton Varda]

No clean solution, but with hacks we've been able to solve the problem for our use case.

I set the heap limit to 8x the limit I actually want. Then, after each call into the isolate, I check if the memory usage has gone over the intended limit. If so, I invoke garbage collection. If it's *still* over the limit after that, then I terminate the isolate at that point.

Meanwhile, we also enforce a CPU time limit of 50ms. In practice, a script that allocates tons of memory tends to run out of CPU time before it can hit the 8x heap limit (especially as the GC slows things down when approaching the limit).

This together seems to solve the case of scripts that accidentally allocate too much memory.

A carefully-crafted malicious script can still cause an abort. In that case, the SIGABRT single handler blacklists the offending script and simply leaves the thread in limbo rather than exit the process. Some time later, if we need that RAM back, we can do a clean restart, in which we start a new process to handle new requests while letting the old process finish up anything in-flight and then exit.

One thing I haven't had time to experiment with yet is whether we can longjmp() out of the signal handler and then delete the isolate, rather than leave it in limbo. Obviously this would be super-leaky, but perhaps we could reclaim most of the resources associated with the isolate. (In my first message in this thread I was contemplating whether I could compile with exceptions enabled in order to be able to do a proper stack unwind invoking destructors, which could perhaps make things a little bit less leaky, even though V8 is not exception-safe. But that won't actually work because libunwind can't unwind through JIT'd code.)

-Kenton

You received this message because you are subscribed to a topic in the Google Groups "v8-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/v8-users/vKn1hVs8KNQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to v8-users+unsubscribe@googlegroups.com.

[Prakash Bailkeri]

Thanks for the clarification.

Prakash

To unsubscribe from this group and all its topics, send an email to v8-users+u...@googlegroups.com.

[ma...@jsonar.com]

To be perfectly honest, this seems rather pointless to terminate the process in this way.

If, in fact, a process were legitimately using too much memory, the underlying operating system should be entirely capable of killing it anyways.  Having v8 do so of its own accord instead of simply returning an error condition that could be detected by an embedded application as an out of memory condition with the v8 engine seems superfluous at best, and completely unusable for many purposes at worst.  

[Kenton Varda]

On Tue, Dec 4, 2018 at 12:40 PM markt via v8-users <v8-u...@googlegroups.com> wrote:

To be perfectly honest, this seems rather pointless to terminate the process in this way.

If, in fact, a process were legitimately using too much memory, the underlying operating system should be entirely capable of killing it anyways.  Having v8 do so of its own accord instead of simply returning an error condition that could be detected by an embedded application as an out of memory condition with the v8 engine seems superfluous at best, and completely unusable for many purposes at worst.

I don't think that's the point. The point is that many code paths in V8 are not prepared to handle memory allocation failures. Gracefully handling allocation errors often requires significantly more code, tests, and general engineering effort, which would be totally wasted for Chrome's use case since Chrome will just terminate the process anyway.

Personally I would much prefer if V8 did handle these cases but it makes plenty of sense why they don't.

-Kenton

 

--

[ma...@jsonar.com]

I was not suggesting that v8 be designed to recover from situations where the native process has actually run out of all available memory, I am suggesting that v8 should be designed to recover (in some way that does not terminate the process) from situations where the javascript code that may be executing is consuming more memory than what the javascript code was permitted to use, which may be *FAR* less than the amount of memory that is actually available on the computer.   The isolate that contains the badly behaved javascript code should be marked as having an irrecoverable error associated with it, and any further attempts to manipulate the isolate should behave as if that isolate has already been terminated.    This irrecoverable error could be detected by the embedder when it resumes control, and the embedder can recover from this situation with respect to the rest of its native process in its own way.  The isolate would remain unusable, but the native process would not crash.  Eventually, the embedder would have to simply dispose of the isolate, and start over if desired with a new one, if more javascript code execution is desired, reporting the out of memory situation as applicable to the situation, perhaps by blacklisting the script which caused the behavior until it can be manually vettted to ensure that the situation does not happen again. 

My point remains, if a process is genuinely consuming too much memory, the operating system will unceremoniously kill the process anyways.  It is redundant at best to deliberately put this kind of logic into v8, and at worst renders v8 unusable for many situations.  

[Kenton Varda]

Yes, everyone gets what you're asking for. The problem is that V8's C++ code commonly allocates from the JavaScript heap. Every place where it does that, the C++ code would need a path to handle allocation failure. It's entirely possible to do, but it's a lot of work and hard to test.

-Kenton

[ma...@jsonar.com]

Actually, no real recovery from such a condition needs to occur in v8 at all.  If the process is genuinely out of memory, the OS will promptly kill it anyways.. all that needs to happen in v8 when the javascript environment tries to allocate more memory than the environment was told it had available is allow the allocation to succeed, but report an out of memory exception, and terminate the isolate.  At the very least, it should be a compile-time option to have this behavior.

[Tudor Bosman]

Bringing this thread back -- I made a separate post about this (https://groups.google.com/g/v8-users/c/ZRjlM_hmdNk).

I'm doing Array(1e7).fill(1) and the near heap limit callback doesn't get called, and the heap limit is not obeyed. I set the heap limit to 4MiB and the tiny little script happily allocates 40.

I don't fully understand when the heap limit is actually checked -- it doesn't seem to happen on every allocation (I suspect that it would be way too expensive).

I'm trying to RequestInterrupt from the near heap limit callback, and, when interrupted, to TerminateExecution. I even tried doing this from a GC epilogue instead of the near limit callback, but that doesn't get called either (there is obviously no garbage from that tiny script).

Thanks,

-Tudor.

[dinf...@chromium.org]

Hi,

The near heap limit callbacks are only called during GC. So I suspect what's happening here is that after a GC live heap objects fit nicely into the heap, so the GC doesn't invoke the callback. The GC doesn't really know about that quite large object allocation. I guess we could invoke that near heap limit callback on allocations as well, you can open an issue for that or provide a patch.

Dominik